Note: This is test shard 1 of 8.
[==========] Running 8 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from AdminCliTest
[ RUN ] AdminCliTest.RebalancerReportOnly
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 14:13:16.001796 19835 test_util.cc:268] Using random seed: 877256145
I20240502 14:13:16.015259 19835 ts_itest-base.cc:112] Starting cluster with:
I20240502 14:13:16.015362 19835 ts_itest-base.cc:113] --------------
I20240502 14:13:16.015410 19835 ts_itest-base.cc:114] 5 tablet servers
I20240502 14:13:16.015487 19835 ts_itest-base.cc:115] 3 replicas per TS
I20240502 14:13:16.015559 19835 ts_itest-base.cc:116] --------------
2024-05-02T14:13:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:13:16Z Disabled control of system clock
I20240502 14:13:16.044543 19835 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_zeJNl/build/asan/bin/kudu
/tmp/dist-test-task_zeJNl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.94.254:34155
--webserver_interface=127.19.94.254
--webserver_port=0
--builtin_ntp_servers=127.19.94.212:37763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.94.254:34155 with env {}
W20240502 14:13:16.430958 19847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:16.431301 19847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:16.431386 19847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:16.440023 19847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:13:16.440160 19847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:16.440207 19847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:13:16.440258 19847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:13:16.451786 19847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.94.212:37763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.94.254:34155
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.94.254:34155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.19.94.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:16.453394 19847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:16.455355 19847 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:16.467996 19852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:16.468641 19853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:16.469039 19855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:16.469472 19847 server_base.cc:1034] running on GCE node
I20240502 14:13:16.470325 19847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:16.471824 19847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:16.473111 19847 hybrid_clock.cc:648] HybridClock initialized: now 1714659196472991 us; error 60 us; skew 500 ppm
I20240502 14:13:16.473508 19847 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:16.477176 19847 webserver.cc:460] Webserver started at http://127.19.94.254:33097/ using document root <none> and password file <none>
I20240502 14:13:16.478099 19847 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:16.478345 19847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:16.478797 19847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:16.482160 19847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "bbcaa2b4d6224ec8914049ca5118a8b3"
format_stamp: "Formatted at 2024-05-02 14:13:16 on dist-test-slave-3j2v"
I20240502 14:13:16.482964 19847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "bbcaa2b4d6224ec8914049ca5118a8b3"
format_stamp: "Formatted at 2024-05-02 14:13:16 on dist-test-slave-3j2v"
I20240502 14:13:16.488679 19847 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20240502 14:13:16.492537 19861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:16.494055 19847 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 14:13:16.494277 19847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/wal
uuid: "bbcaa2b4d6224ec8914049ca5118a8b3"
format_stamp: "Formatted at 2024-05-02 14:13:16 on dist-test-slave-3j2v"
I20240502 14:13:16.494508 19847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:16.519508 19847 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:16.520537 19847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:16.520907 19847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:16.545915 19847 rpc_server.cc:307] RPC server started. Bound to: 127.19.94.254:34155
I20240502 14:13:16.545966 19912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.94.254:34155 every 8 connection(s)
I20240502 14:13:16.547353 19847 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 14:13:16.551365 19913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:16.555874 19835 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_zeJNl/build/asan/bin/kudu as pid 19847
I20240502 14:13:16.556193 19835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 14:13:16.563982 19913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbcaa2b4d6224ec8914049ca5118a8b3: Bootstrap starting.
I20240502 14:13:16.568773 19913 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbcaa2b4d6224ec8914049ca5118a8b3: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:16.570210 19913 log.cc:824] T 00000000000000000000000000000000 P bbcaa2b4d6224ec8914049ca5118a8b3: Log is configured to *not* fsync() on all Append() calls
I20240502 14:13:16.573809 19913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbcaa2b4d6224ec8914049ca5118a8b3: No bootstrap required, opened a new log
I20240502 14:13:16.585517 19913 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbcaa2b4d6224ec8914049ca5118a8b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbcaa2b4d6224ec8914049ca5118a8b3" member_type: VOTER last_known_addr { host: "127.19.94.254" port: 34155 } }
I20240502 14:13:16.586040 19913 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbcaa2b4d6224ec8914049ca5118a8b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:16.586164 19913 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbcaa2b4d6224ec8914049ca5118a8b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbcaa2b4d6224ec8914049ca5118a8b3, State: Initialized, Role: FOLLOWER
I20240502 14:13:16.586989 19913 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbcaa2b4d6224ec8914049ca5118a8b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbcaa2b4d6224ec8914049ca5118a8b3" member_type: VOTER last_known_addr { host: "127.19.94.254" port: 34155 } }
I20240502 14:13:16.587306 19913 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbcaa2b4d6224ec8914049ca5118a8b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:16.587474 19913 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbcaa2b4d6224ec8914049ca5118a8b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:16.587816 19913 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbcaa2b4d6224ec8914049ca5118a8b3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:16.590883 19913 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbcaa2b4d6224ec8914049ca5118a8b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbcaa2b4d6224ec8914049ca5118a8b3" member_type: VOTER last_known_addr { host: "127.19.94.254" port: 34155 } }
I20240502 14:13:16.591562 19913 leader_election.cc:302] T 00000000000000000000000000000000 P bbcaa2b4d6224ec8914049ca5118a8b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbcaa2b4d6224ec8914049ca5118a8b3; no voters:
I20240502 14:13:16.592814 19918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbcaa2b4d6224ec8914049ca5118a8b3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:16.593261 19913 leader_election.cc:288] T 00000000000000000000000000000000 P bbcaa2b4d6224ec8914049ca5118a8b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:16.594796 19918 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbcaa2b4d6224ec8914049ca5118a8b3 [term 1 LEADER]: Becoming Leader. State: Replica: bbcaa2b4d6224ec8914049ca5118a8b3, State: Running, Role: LEADER
I20240502 14:13:16.595571 19918 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbcaa2b4d6224ec8914049ca5118a8b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbcaa2b4d6224ec8914049ca5118a8b3" member_type: VOTER last_known_addr { host: "127.19.94.254" port: 34155 } }
I20240502 14:13:16.596896 19913 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbcaa2b4d6224ec8914049ca5118a8b3 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:13:16.599012 19920 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbcaa2b4d6224ec8914049ca5118a8b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbcaa2b4d6224ec8914049ca5118a8b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbcaa2b4d6224ec8914049ca5118a8b3" member_type: VOTER last_known_addr { host: "127.19.94.254" port: 34155 } } }
I20240502 14:13:16.599355 19920 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbcaa2b4d6224ec8914049ca5118a8b3 [sys.catalog]: This master's current role is: LEADER
I20240502 14:13:16.599169 19919 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbcaa2b4d6224ec8914049ca5118a8b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbcaa2b4d6224ec8914049ca5118a8b3. Latest consensus state: current_term: 1 leader_uuid: "bbcaa2b4d6224ec8914049ca5118a8b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbcaa2b4d6224ec8914049ca5118a8b3" member_type: VOTER last_known_addr { host: "127.19.94.254" port: 34155 } } }
I20240502 14:13:16.599686 19919 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbcaa2b4d6224ec8914049ca5118a8b3 [sys.catalog]: This master's current role is: LEADER
I20240502 14:13:16.606108 19923 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:13:16.614037 19923 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:13:16.627379 19923 catalog_manager.cc:1347] Generated new cluster ID: b98d27710e8744ef95718710e33f4420
I20240502 14:13:16.627636 19923 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:13:16.676491 19923 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:13:16.677898 19923 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:13:16.687142 19923 catalog_manager.cc:5857] T 00000000000000000000000000000000 P bbcaa2b4d6224ec8914049ca5118a8b3: Generated new TSK 0
I20240502 14:13:16.688324 19923 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:13:16.696822 19835 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_zeJNl/build/asan/bin/kudu
/tmp/dist-test-task_zeJNl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.94.193:0
--local_ip_for_outbound_sockets=127.19.94.193
--webserver_interface=127.19.94.193
--webserver_port=0
--tserver_master_addrs=127.19.94.254:34155
--builtin_ntp_servers=127.19.94.212:37763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20240502 14:13:17.112577 19937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:17.112905 19937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:17.113008 19937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:17.121042 19937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:17.121237 19937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.94.193
I20240502 14:13:17.130918 19937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.94.212:37763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.94.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.19.94.193
--webserver_port=0
--tserver_master_addrs=127.19.94.254:34155
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.94.193
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:17.132284 19937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:17.134182 19937 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:17.146886 19942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:17.147377 19945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:17.148154 19943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:17.148869 19937 server_base.cc:1034] running on GCE node
I20240502 14:13:17.149713 19937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:17.150975 19937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:17.152264 19937 hybrid_clock.cc:648] HybridClock initialized: now 1714659197152184 us; error 44 us; skew 500 ppm
I20240502 14:13:17.152686 19937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:17.155146 19937 webserver.cc:460] Webserver started at http://127.19.94.193:43639/ using document root <none> and password file <none>
I20240502 14:13:17.156090 19937 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:17.156246 19937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:17.156697 19937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:17.159562 19937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "92b2b68f2d614b8da92724c48432ecae"
format_stamp: "Formatted at 2024-05-02 14:13:17 on dist-test-slave-3j2v"
I20240502 14:13:17.160444 19937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "92b2b68f2d614b8da92724c48432ecae"
format_stamp: "Formatted at 2024-05-02 14:13:17 on dist-test-slave-3j2v"
I20240502 14:13:17.166330 19937 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240502 14:13:17.170303 19951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:17.171936 19937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240502 14:13:17.172161 19937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "92b2b68f2d614b8da92724c48432ecae"
format_stamp: "Formatted at 2024-05-02 14:13:17 on dist-test-slave-3j2v"
I20240502 14:13:17.172379 19937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:17.229285 19937 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:17.230253 19937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:17.230588 19937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:17.231684 19937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:17.233403 19937 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:17.233494 19937 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:17.233630 19937 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:17.233687 19937 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:17.278568 19937 rpc_server.cc:307] RPC server started. Bound to: 127.19.94.193:41249
I20240502 14:13:17.278645 20063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.94.193:41249 every 8 connection(s)
I20240502 14:13:17.279949 19937 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 14:13:17.283531 19835 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_zeJNl/build/asan/bin/kudu as pid 19937
I20240502 14:13:17.283758 19835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 14:13:17.287333 19835 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_zeJNl/build/asan/bin/kudu
/tmp/dist-test-task_zeJNl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.94.194:0
--local_ip_for_outbound_sockets=127.19.94.194
--webserver_interface=127.19.94.194
--webserver_port=0
--tserver_master_addrs=127.19.94.254:34155
--builtin_ntp_servers=127.19.94.212:37763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240502 14:13:17.294776 20064 heartbeater.cc:347] Connected to a master server at 127.19.94.254:34155
I20240502 14:13:17.295090 20064 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:17.295964 20064 heartbeater.cc:511] Master 127.19.94.254:34155 requested a full tablet report, sending...
I20240502 14:13:17.298318 19878 ts_manager.cc:194] Registered new tserver with Master: 92b2b68f2d614b8da92724c48432ecae (127.19.94.193:41249)
I20240502 14:13:17.300105 19878 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.94.193:35743
W20240502 14:13:17.649981 20068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:17.650341 20068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:17.650497 20068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:17.658452 20068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:17.658713 20068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.94.194
I20240502 14:13:17.670241 20068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.94.212:37763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.94.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.19.94.194
--webserver_port=0
--tserver_master_addrs=127.19.94.254:34155
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.94.194
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:17.671763 20068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:17.673717 20068 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:17.687847 20073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:17.688557 20074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:17.689033 20076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:17.689832 20068 server_base.cc:1034] running on GCE node
I20240502 14:13:17.690517 20068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:17.691557 20068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:17.692804 20068 hybrid_clock.cc:648] HybridClock initialized: now 1714659197692735 us; error 46 us; skew 500 ppm
I20240502 14:13:17.693145 20068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:17.695631 20068 webserver.cc:460] Webserver started at http://127.19.94.194:45309/ using document root <none> and password file <none>
I20240502 14:13:17.696426 20068 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:17.696633 20068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:17.697010 20068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:17.700126 20068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "44e3b4d78f8b46d1b563d8526a4bae03"
format_stamp: "Formatted at 2024-05-02 14:13:17 on dist-test-slave-3j2v"
I20240502 14:13:17.701151 20068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "44e3b4d78f8b46d1b563d8526a4bae03"
format_stamp: "Formatted at 2024-05-02 14:13:17 on dist-test-slave-3j2v"
I20240502 14:13:17.707862 20068 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20240502 14:13:17.711899 20082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:17.713486 20068 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:13:17.713671 20068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "44e3b4d78f8b46d1b563d8526a4bae03"
format_stamp: "Formatted at 2024-05-02 14:13:17 on dist-test-slave-3j2v"
I20240502 14:13:17.713877 20068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:17.729457 20068 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:17.730425 20068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:17.730741 20068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:17.731734 20068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:17.733330 20068 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:17.733433 20068 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:17.733592 20068 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:17.733649 20068 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:17.770526 20068 rpc_server.cc:307] RPC server started. Bound to: 127.19.94.194:36157
I20240502 14:13:17.770510 20194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.94.194:36157 every 8 connection(s)
I20240502 14:13:17.772068 20068 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 14:13:17.778843 19835 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_zeJNl/build/asan/bin/kudu as pid 20068
I20240502 14:13:17.779080 19835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 14:13:17.782737 19835 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_zeJNl/build/asan/bin/kudu
/tmp/dist-test-task_zeJNl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.94.195:0
--local_ip_for_outbound_sockets=127.19.94.195
--webserver_interface=127.19.94.195
--webserver_port=0
--tserver_master_addrs=127.19.94.254:34155
--builtin_ntp_servers=127.19.94.212:37763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240502 14:13:17.788477 20195 heartbeater.cc:347] Connected to a master server at 127.19.94.254:34155
I20240502 14:13:17.789176 20195 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:17.790364 20195 heartbeater.cc:511] Master 127.19.94.254:34155 requested a full tablet report, sending...
I20240502 14:13:17.792213 19878 ts_manager.cc:194] Registered new tserver with Master: 44e3b4d78f8b46d1b563d8526a4bae03 (127.19.94.194:36157)
I20240502 14:13:17.793082 19878 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.94.194:60963
W20240502 14:13:18.179104 20198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:18.179437 20198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:18.179630 20198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:18.187875 20198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:18.188105 20198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.94.195
I20240502 14:13:18.197997 20198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.94.212:37763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.94.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.19.94.195
--webserver_port=0
--tserver_master_addrs=127.19.94.254:34155
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.94.195
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:18.199441 20198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:18.201331 20198 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:18.215523 20203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:18.216181 20204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:18.216959 20206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:18.217175 20198 server_base.cc:1034] running on GCE node
I20240502 14:13:18.217861 20198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:18.219004 20198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:18.220252 20198 hybrid_clock.cc:648] HybridClock initialized: now 1714659198220175 us; error 49 us; skew 500 ppm
I20240502 14:13:18.220633 20198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:18.223356 20198 webserver.cc:460] Webserver started at http://127.19.94.195:40157/ using document root <none> and password file <none>
I20240502 14:13:18.224171 20198 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:18.224318 20198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:18.224690 20198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:18.227563 20198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "a4fa37bd29574001a270cd939b687084"
format_stamp: "Formatted at 2024-05-02 14:13:18 on dist-test-slave-3j2v"
I20240502 14:13:18.228371 20198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "a4fa37bd29574001a270cd939b687084"
format_stamp: "Formatted at 2024-05-02 14:13:18 on dist-test-slave-3j2v"
I20240502 14:13:18.234051 20198 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240502 14:13:18.237826 20212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:18.239434 20198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 14:13:18.239625 20198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "a4fa37bd29574001a270cd939b687084"
format_stamp: "Formatted at 2024-05-02 14:13:18 on dist-test-slave-3j2v"
I20240502 14:13:18.239835 20198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:18.257596 20198 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:18.258523 20198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:18.258846 20198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:18.259987 20198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:18.261874 20198 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:18.261962 20198 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:18.262105 20198 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:18.262166 20198 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:18.301903 20198 rpc_server.cc:307] RPC server started. Bound to: 127.19.94.195:42111
I20240502 14:13:18.301936 20324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.94.195:42111 every 8 connection(s)
I20240502 14:13:18.303185 20064 heartbeater.cc:503] Master 127.19.94.254:34155 was elected leader, sending a full tablet report...
I20240502 14:13:18.303303 20198 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 14:13:18.305887 19835 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_zeJNl/build/asan/bin/kudu as pid 20198
I20240502 14:13:18.306123 19835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 14:13:18.309509 19835 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_zeJNl/build/asan/bin/kudu
/tmp/dist-test-task_zeJNl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.94.196:0
--local_ip_for_outbound_sockets=127.19.94.196
--webserver_interface=127.19.94.196
--webserver_port=0
--tserver_master_addrs=127.19.94.254:34155
--builtin_ntp_servers=127.19.94.212:37763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240502 14:13:18.319090 20325 heartbeater.cc:347] Connected to a master server at 127.19.94.254:34155
I20240502 14:13:18.319442 20325 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:18.320464 20325 heartbeater.cc:511] Master 127.19.94.254:34155 requested a full tablet report, sending...
I20240502 14:13:18.321992 19878 ts_manager.cc:194] Registered new tserver with Master: a4fa37bd29574001a270cd939b687084 (127.19.94.195:42111)
I20240502 14:13:18.322835 19878 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.94.195:51367
W20240502 14:13:18.701092 20329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:18.701396 20329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:18.701560 20329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:18.709821 20329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:18.710084 20329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.94.196
I20240502 14:13:18.719754 20329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.94.212:37763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.94.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.19.94.196
--webserver_port=0
--tserver_master_addrs=127.19.94.254:34155
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.94.196
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:18.721174 20329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:18.723367 20329 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:18.737293 20334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:18.737854 20337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:18.738564 20335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:18.739130 20329 server_base.cc:1034] running on GCE node
I20240502 14:13:18.739845 20329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:18.740949 20329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:18.742197 20329 hybrid_clock.cc:648] HybridClock initialized: now 1714659198742108 us; error 52 us; skew 500 ppm
I20240502 14:13:18.742537 20329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:18.745020 20329 webserver.cc:460] Webserver started at http://127.19.94.196:44427/ using document root <none> and password file <none>
I20240502 14:13:18.745846 20329 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:18.745968 20329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:18.746299 20329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:18.749151 20329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "8601e3e3bfc542dc9e0835f7e5260230"
format_stamp: "Formatted at 2024-05-02 14:13:18 on dist-test-slave-3j2v"
I20240502 14:13:18.749989 20329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "8601e3e3bfc542dc9e0835f7e5260230"
format_stamp: "Formatted at 2024-05-02 14:13:18 on dist-test-slave-3j2v"
I20240502 14:13:18.755748 20329 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240502 14:13:18.759522 20343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:18.761130 20329 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 14:13:18.761287 20329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "8601e3e3bfc542dc9e0835f7e5260230"
format_stamp: "Formatted at 2024-05-02 14:13:18 on dist-test-slave-3j2v"
I20240502 14:13:18.761476 20329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:18.787917 20329 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:18.788957 20329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:18.789274 20329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:18.790302 20329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:18.792080 20329 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:18.792169 20329 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:18.792281 20329 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:18.792328 20329 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:18.796908 20195 heartbeater.cc:503] Master 127.19.94.254:34155 was elected leader, sending a full tablet report...
I20240502 14:13:18.834281 20329 rpc_server.cc:307] RPC server started. Bound to: 127.19.94.196:34793
I20240502 14:13:18.834317 20455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.94.196:34793 every 8 connection(s)
I20240502 14:13:18.835774 20329 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 14:13:18.841884 19835 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_zeJNl/build/asan/bin/kudu as pid 20329
I20240502 14:13:18.842063 19835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 14:13:18.845744 19835 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_zeJNl/build/asan/bin/kudu
/tmp/dist-test-task_zeJNl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.94.197:0
--local_ip_for_outbound_sockets=127.19.94.197
--webserver_interface=127.19.94.197
--webserver_port=0
--tserver_master_addrs=127.19.94.254:34155
--builtin_ntp_servers=127.19.94.212:37763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240502 14:13:18.852751 20456 heartbeater.cc:347] Connected to a master server at 127.19.94.254:34155
I20240502 14:13:18.853101 20456 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:18.854007 20456 heartbeater.cc:511] Master 127.19.94.254:34155 requested a full tablet report, sending...
I20240502 14:13:18.855648 19878 ts_manager.cc:194] Registered new tserver with Master: 8601e3e3bfc542dc9e0835f7e5260230 (127.19.94.196:34793)
I20240502 14:13:18.856544 19878 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.94.196:52097
W20240502 14:13:19.256860 20460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:19.257203 20460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:19.257373 20460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:19.265389 20460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:19.265653 20460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.94.197
I20240502 14:13:19.275759 20460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.94.212:37763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.94.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.19.94.197
--webserver_port=0
--tserver_master_addrs=127.19.94.254:34155
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.94.197
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:19.277305 20460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:19.279609 20460 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:19.292207 20465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:19.292896 20466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:19.293591 20468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:19.293751 20460 server_base.cc:1034] running on GCE node
I20240502 14:13:19.294413 20460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:19.295418 20460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:19.296691 20460 hybrid_clock.cc:648] HybridClock initialized: now 1714659199296622 us; error 41 us; skew 500 ppm
I20240502 14:13:19.297088 20460 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:19.300125 20460 webserver.cc:460] Webserver started at http://127.19.94.197:43661/ using document root <none> and password file <none>
I20240502 14:13:19.301192 20460 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:19.301349 20460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:19.301712 20460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:19.304576 20460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "e03ab67388564eefbba725a21c63a067"
format_stamp: "Formatted at 2024-05-02 14:13:19 on dist-test-slave-3j2v"
I20240502 14:13:19.305426 20460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "e03ab67388564eefbba725a21c63a067"
format_stamp: "Formatted at 2024-05-02 14:13:19 on dist-test-slave-3j2v"
I20240502 14:13:19.314488 20460 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.006s sys 0.001s
I20240502 14:13:19.319031 20474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:19.320695 20460 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:13:19.320881 20460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "e03ab67388564eefbba725a21c63a067"
format_stamp: "Formatted at 2024-05-02 14:13:19 on dist-test-slave-3j2v"
I20240502 14:13:19.321089 20460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:19.325585 20325 heartbeater.cc:503] Master 127.19.94.254:34155 was elected leader, sending a full tablet report...
I20240502 14:13:19.350493 20460 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:19.351512 20460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:19.352059 20460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:19.353124 20460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:19.354697 20460 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:19.354774 20460 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:19.354868 20460 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:19.354913 20460 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:19.396745 20460 rpc_server.cc:307] RPC server started. Bound to: 127.19.94.197:37427
I20240502 14:13:19.396806 20586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.94.197:37427 every 8 connection(s)
I20240502 14:13:19.398166 20460 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 14:13:19.399477 19835 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_zeJNl/build/asan/bin/kudu as pid 20460
I20240502 14:13:19.399693 19835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 14:13:19.413210 20587 heartbeater.cc:347] Connected to a master server at 127.19.94.254:34155
I20240502 14:13:19.413517 20587 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:19.414359 20587 heartbeater.cc:511] Master 127.19.94.254:34155 requested a full tablet report, sending...
I20240502 14:13:19.415948 19878 ts_manager.cc:194] Registered new tserver with Master: e03ab67388564eefbba725a21c63a067 (127.19.94.197:37427)
I20240502 14:13:19.417101 19878 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.94.197:34869
I20240502 14:13:19.417460 19835 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 14:13:19.442164 19878 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52844:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 14:13:19.486707 20260 tablet_service.cc:1467] Processing CreateTablet for tablet c7db06d8190f431388da2f53885fbf46 (DEFAULT_TABLE table=TestTable [id=f41c5f0303bf432aab7a7924d3603247]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:13:19.488462 20260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7db06d8190f431388da2f53885fbf46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:19.491580 19999 tablet_service.cc:1467] Processing CreateTablet for tablet c7db06d8190f431388da2f53885fbf46 (DEFAULT_TABLE table=TestTable [id=f41c5f0303bf432aab7a7924d3603247]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:13:19.491891 20391 tablet_service.cc:1467] Processing CreateTablet for tablet c7db06d8190f431388da2f53885fbf46 (DEFAULT_TABLE table=TestTable [id=f41c5f0303bf432aab7a7924d3603247]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:13:19.493208 19999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7db06d8190f431388da2f53885fbf46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:19.493353 20391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7db06d8190f431388da2f53885fbf46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:19.504499 20606 tablet_bootstrap.cc:492] T c7db06d8190f431388da2f53885fbf46 P a4fa37bd29574001a270cd939b687084: Bootstrap starting.
I20240502 14:13:19.507397 20607 tablet_bootstrap.cc:492] T c7db06d8190f431388da2f53885fbf46 P 92b2b68f2d614b8da92724c48432ecae: Bootstrap starting.
I20240502 14:13:19.508843 20606 tablet_bootstrap.cc:654] T c7db06d8190f431388da2f53885fbf46 P a4fa37bd29574001a270cd939b687084: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:19.509192 20608 tablet_bootstrap.cc:492] T c7db06d8190f431388da2f53885fbf46 P 8601e3e3bfc542dc9e0835f7e5260230: Bootstrap starting.
I20240502 14:13:19.511245 20607 tablet_bootstrap.cc:654] T c7db06d8190f431388da2f53885fbf46 P 92b2b68f2d614b8da92724c48432ecae: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:19.511591 20606 log.cc:824] T c7db06d8190f431388da2f53885fbf46 P a4fa37bd29574001a270cd939b687084: Log is configured to *not* fsync() on all Append() calls
I20240502 14:13:19.512327 20607 log.cc:824] T c7db06d8190f431388da2f53885fbf46 P 92b2b68f2d614b8da92724c48432ecae: Log is configured to *not* fsync() on all Append() calls
I20240502 14:13:19.513127 20608 tablet_bootstrap.cc:654] T c7db06d8190f431388da2f53885fbf46 P 8601e3e3bfc542dc9e0835f7e5260230: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:19.514328 20608 log.cc:824] T c7db06d8190f431388da2f53885fbf46 P 8601e3e3bfc542dc9e0835f7e5260230: Log is configured to *not* fsync() on all Append() calls
I20240502 14:13:19.515184 20606 tablet_bootstrap.cc:492] T c7db06d8190f431388da2f53885fbf46 P a4fa37bd29574001a270cd939b687084: No bootstrap required, opened a new log
I20240502 14:13:19.515460 20606 ts_tablet_manager.cc:1367] T c7db06d8190f431388da2f53885fbf46 P a4fa37bd29574001a270cd939b687084: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20240502 14:13:19.517503 20608 tablet_bootstrap.cc:492] T c7db06d8190f431388da2f53885fbf46 P 8601e3e3bfc542dc9e0835f7e5260230: No bootstrap required, opened a new log
I20240502 14:13:19.517879 20608 ts_tablet_manager.cc:1367] T c7db06d8190f431388da2f53885fbf46 P 8601e3e3bfc542dc9e0835f7e5260230: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20240502 14:13:19.522348 20607 tablet_bootstrap.cc:492] T c7db06d8190f431388da2f53885fbf46 P 92b2b68f2d614b8da92724c48432ecae: No bootstrap required, opened a new log
I20240502 14:13:19.522679 20607 ts_tablet_manager.cc:1367] T c7db06d8190f431388da2f53885fbf46 P 92b2b68f2d614b8da92724c48432ecae: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.005s
I20240502 14:13:19.526208 20606 raft_consensus.cc:348] T c7db06d8190f431388da2f53885fbf46 P a4fa37bd29574001a270cd939b687084 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4fa37bd29574001a270cd939b687084" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 42111 } } peers { permanent_uuid: "8601e3e3bfc542dc9e0835f7e5260230" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 34793 } } peers { permanent_uuid: "92b2b68f2d614b8da92724c48432ecae" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 41249 } }
I20240502 14:13:19.526782 20606 raft_consensus.cc:374] T c7db06d8190f431388da2f53885fbf46 P a4fa37bd29574001a270cd939b687084 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:19.526896 20606 raft_consensus.cc:729] T c7db06d8190f431388da2f53885fbf46 P a4fa37bd29574001a270cd939b687084 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4fa37bd29574001a270cd939b687084, State: Initialized, Role: FOLLOWER
I20240502 14:13:19.527509 20608 raft_consensus.cc:348] T c7db06d8190f431388da2f53885fbf46 P 8601e3e3bfc542dc9e0835f7e5260230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4fa37bd29574001a270cd939b687084" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 42111 } } peers { permanent_uuid: "8601e3e3bfc542dc9e0835f7e5260230" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 34793 } } peers { permanent_uuid: "92b2b68f2d614b8da92724c48432ecae" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 41249 } }
I20240502 14:13:19.527693 20606 consensus_queue.cc:260] T c7db06d8190f431388da2f53885fbf46 P a4fa37bd29574001a270cd939b687084 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4fa37bd29574001a270cd939b687084" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 42111 } } peers { permanent_uuid: "8601e3e3bfc542dc9e0835f7e5260230" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 34793 } } peers { permanent_uuid: "92b2b68f2d614b8da92724c48432ecae" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 41249 } }
I20240502 14:13:19.527977 20608 raft_consensus.cc:374] T c7db06d8190f431388da2f53885fbf46 P 8601e3e3bfc542dc9e0835f7e5260230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:19.528108 20608 raft_consensus.cc:729] T c7db06d8190f431388da2f53885fbf46 P 8601e3e3bfc542dc9e0835f7e5260230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8601e3e3bfc542dc9e0835f7e5260230, State: Initialized, Role: FOLLOWER
I20240502 14:13:19.529357 20608 consensus_queue.cc:260] T c7db06d8190f431388da2f53885fbf46 P 8601e3e3bfc542dc9e0835f7e5260230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4fa37bd29574001a270cd939b687084" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 42111 } } peers { permanent_uuid: "8601e3e3bfc542dc9e0835f7e5260230" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 34793 } } peers { permanent_uuid: "92b2b68f2d614b8da92724c48432ecae" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 41249 } }
I20240502 14:13:19.529724 20606 ts_tablet_manager.cc:1398] T c7db06d8190f431388da2f53885fbf46 P a4fa37bd29574001a270cd939b687084: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20240502 14:13:19.530795 20456 heartbeater.cc:503] Master 127.19.94.254:34155 was elected leader, sending a full tablet report...
I20240502 14:13:19.530550 20607 raft_consensus.cc:348] T c7db06d8190f431388da2f53885fbf46 P 92b2b68f2d614b8da92724c48432ecae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4fa37bd29574001a270cd939b687084" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 42111 } } peers { permanent_uuid: "8601e3e3bfc542dc9e0835f7e5260230" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 34793 } } peers { permanent_uuid: "92b2b68f2d614b8da92724c48432ecae" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 41249 } }
I20240502 14:13:19.531033 20607 raft_consensus.cc:374] T c7db06d8190f431388da2f53885fbf46 P 92b2b68f2d614b8da92724c48432ecae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:19.531134 20607 raft_consensus.cc:729] T c7db06d8190f431388da2f53885fbf46 P 92b2b68f2d614b8da92724c48432ecae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92b2b68f2d614b8da92724c48432ecae, State: Initialized, Role: FOLLOWER
I20240502 14:13:19.531847 20607 consensus_queue.cc:260] T c7db06d8190f431388da2f53885fbf46 P 92b2b68f2d614b8da92724c48432ecae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4fa37bd29574001a270cd939b687084" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 42111 } } peers { permanent_uuid: "8601e3e3bfc542dc9e0835f7e5260230" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 34793 } } peers { permanent_uuid: "92b2b68f2d614b8da92724c48432ecae" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 41249 } }
I20240502 14:13:19.533591 20608 ts_tablet_manager.cc:1398] T c7db06d8190f431388da2f53885fbf46 P 8601e3e3bfc542dc9e0835f7e5260230: Time spent starting tablet: real 0.016s user 0.012s sys 0.000s
I20240502 14:13:19.535485 20607 ts_tablet_manager.cc:1398] T c7db06d8190f431388da2f53885fbf46 P 92b2b68f2d614b8da92724c48432ecae: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
W20240502 14:13:19.556190 20326 tablet.cc:2369] T c7db06d8190f431388da2f53885fbf46 P a4fa37bd29574001a270cd939b687084: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 14:13:19.588517 20457 tablet.cc:2369] T c7db06d8190f431388da2f53885fbf46 P 8601e3e3bfc542dc9e0835f7e5260230: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 14:13:19.783437 20065 tablet.cc:2369] T c7db06d8190f431388da2f53885fbf46 P 92b2b68f2d614b8da92724c48432ecae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:13:19.859097 20612 raft_consensus.cc:482] T c7db06d8190f431388da2f53885fbf46 P a4fa37bd29574001a270cd939b687084 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:13:19.859369 20612 raft_consensus.cc:504] T c7db06d8190f431388da2f53885fbf46 P a4fa37bd29574001a270cd939b687084 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4fa37bd29574001a270cd939b687084" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 42111 } } peers { permanent_uuid: "8601e3e3bfc542dc9e0835f7e5260230" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 34793 } } peers { permanent_uuid: "92b2b68f2d614b8da92724c48432ecae" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 41249 } }
I20240502 14:13:19.861223 20612 leader_election.cc:288] T c7db06d8190f431388da2f53885fbf46 P a4fa37bd29574001a270cd939b687084 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8601e3e3bfc542dc9e0835f7e5260230 (127.19.94.196:34793), 92b2b68f2d614b8da92724c48432ecae (127.19.94.193:41249)
I20240502 14:13:19.869458 20019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7db06d8190f431388da2f53885fbf46" candidate_uuid: "a4fa37bd29574001a270cd939b687084" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92b2b68f2d614b8da92724c48432ecae" is_pre_election: true
I20240502 14:13:19.869509 20411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7db06d8190f431388da2f53885fbf46" candidate_uuid: "a4fa37bd29574001a270cd939b687084" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8601e3e3bfc542dc9e0835f7e5260230" is_pre_election: true
I20240502 14:13:19.870007 20411 raft_consensus.cc:2454] T c7db06d8190f431388da2f53885fbf46 P 8601e3e3bfc542dc9e0835f7e5260230 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4fa37bd29574001a270cd939b687084 in term 0.
I20240502 14:13:19.869973 20019 raft_consensus.cc:2454] T c7db06d8190f431388da2f53885fbf46 P 92b2b68f2d614b8da92724c48432ecae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4fa37bd29574001a270cd939b687084 in term 0.
I20240502 14:13:19.870700 20214 leader_election.cc:302] T c7db06d8190f431388da2f53885fbf46 P a4fa37bd29574001a270cd939b687084 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8601e3e3bfc542dc9e0835f7e5260230, a4fa37bd29574001a270cd939b687084; no voters:
I20240502 14:13:19.871075 20612 raft_consensus.cc:2781] T c7db06d8190f431388da2f53885fbf46 P a4fa37bd29574001a270cd939b687084 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:13:19.871237 20612 raft_consensus.cc:482] T c7db06d8190f431388da2f53885fbf46 P a4fa37bd29574001a270cd939b687084 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:13:19.871352 20612 raft_consensus.cc:3037] T c7db06d8190f431388da2f53885fbf46 P a4fa37bd29574001a270cd939b687084 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:19.873661 20612 raft_consensus.cc:504] T c7db06d8190f431388da2f53885fbf46 P a4fa37bd29574001a270cd939b687084 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4fa37bd29574001a270cd939b687084" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 42111 } } peers { permanent_uuid: "8601e3e3bfc542dc9e0835f7e5260230" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 34793 } } peers { permanent_uuid: "92b2b68f2d614b8da92724c48432ecae" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 41249 } }
I20240502 14:13:19.874301 20612 leader_election.cc:288] T c7db06d8190f431388da2f53885fbf46 P a4fa37bd29574001a270cd939b687084 [CANDIDATE]: Term 1 election: Requested vote from peers 8601e3e3bfc542dc9e0835f7e5260230 (127.19.94.196:34793), 92b2b68f2d614b8da92724c48432ecae (127.19.94.193:41249)
I20240502 14:13:19.874939 20411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7db06d8190f431388da2f53885fbf46" candidate_uuid: "a4fa37bd29574001a270cd939b687084" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8601e3e3bfc542dc9e0835f7e5260230"
I20240502 14:13:19.874987 20019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7db06d8190f431388da2f53885fbf46" candidate_uuid: "a4fa37bd29574001a270cd939b687084" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92b2b68f2d614b8da92724c48432ecae"
I20240502 14:13:19.875195 20019 raft_consensus.cc:3037] T c7db06d8190f431388da2f53885fbf46 P 92b2b68f2d614b8da92724c48432ecae [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:19.875195 20411 raft_consensus.cc:3037] T c7db06d8190f431388da2f53885fbf46 P 8601e3e3bfc542dc9e0835f7e5260230 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:19.878545 20019 raft_consensus.cc:2454] T c7db06d8190f431388da2f53885fbf46 P 92b2b68f2d614b8da92724c48432ecae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4fa37bd29574001a270cd939b687084 in term 1.
I20240502 14:13:19.878880 20411 raft_consensus.cc:2454] T c7db06d8190f431388da2f53885fbf46 P 8601e3e3bfc542dc9e0835f7e5260230 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4fa37bd29574001a270cd939b687084 in term 1.
I20240502 14:13:19.879102 20213 leader_election.cc:302] T c7db06d8190f431388da2f53885fbf46 P a4fa37bd29574001a270cd939b687084 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92b2b68f2d614b8da92724c48432ecae, a4fa37bd29574001a270cd939b687084; no voters:
I20240502 14:13:19.879436 20612 raft_consensus.cc:2781] T c7db06d8190f431388da2f53885fbf46 P a4fa37bd29574001a270cd939b687084 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:19.879900 20612 raft_consensus.cc:686] T c7db06d8190f431388da2f53885fbf46 P a4fa37bd29574001a270cd939b687084 [term 1 LEADER]: Becoming Leader. State: Replica: a4fa37bd29574001a270cd939b687084, State: Running, Role: LEADER
I20240502 14:13:19.880498 20612 consensus_queue.cc:237] T c7db06d8190f431388da2f53885fbf46 P a4fa37bd29574001a270cd939b687084 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4fa37bd29574001a270cd939b687084" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 42111 } } peers { permanent_uuid: "8601e3e3bfc542dc9e0835f7e5260230" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 34793 } } peers { permanent_uuid: "92b2b68f2d614b8da92724c48432ecae" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 41249 } }
I20240502 14:13:19.885207 19878 catalog_manager.cc:5484] T c7db06d8190f431388da2f53885fbf46 P a4fa37bd29574001a270cd939b687084 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4fa37bd29574001a270cd939b687084 (127.19.94.195). New cstate: current_term: 1 leader_uuid: "a4fa37bd29574001a270cd939b687084" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4fa37bd29574001a270cd939b687084" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 42111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8601e3e3bfc542dc9e0835f7e5260230" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 34793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92b2b68f2d614b8da92724c48432ecae" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 41249 } health_report { overall_health: UNKNOWN } } }
I20240502 14:13:19.948181 19835 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 14:13:19.951043 19835 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 92b2b68f2d614b8da92724c48432ecae to finish bootstrapping
I20240502 14:13:19.960345 19835 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 44e3b4d78f8b46d1b563d8526a4bae03 to finish bootstrapping
I20240502 14:13:19.969626 19835 ts_itest-base.cc:233] Waiting for 1 tablets on tserver a4fa37bd29574001a270cd939b687084 to finish bootstrapping
I20240502 14:13:19.977624 19835 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 8601e3e3bfc542dc9e0835f7e5260230 to finish bootstrapping
I20240502 14:13:19.985697 19835 ts_itest-base.cc:233] Waiting for 0 tablets on tserver e03ab67388564eefbba725a21c63a067 to finish bootstrapping
I20240502 14:13:20.267042 20612 consensus_queue.cc:1035] T c7db06d8190f431388da2f53885fbf46 P a4fa37bd29574001a270cd939b687084 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8601e3e3bfc542dc9e0835f7e5260230" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 34793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:13:20.277254 20617 consensus_queue.cc:1035] T c7db06d8190f431388da2f53885fbf46 P a4fa37bd29574001a270cd939b687084 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b2b68f2d614b8da92724c48432ecae" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 41249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:13:20.420372 20587 heartbeater.cc:503] Master 127.19.94.254:34155 was elected leader, sending a full tablet report...
I20240502 14:13:20.537765 20130 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:20.541280 20391 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:20.555425 20260 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:20.558126 19999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:20.570891 20522 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:20.748787 20130 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:20.754114 19999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:20.755872 20260 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:20.760632 20522 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:20.761688 20391 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:20.936770 19835 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_zeJNl/build/asan/bin/kudu pid 19937
I20240502 14:13:20.938230 20058 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:13:21.029929 19835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_zeJNl/build/asan/bin/kudu with pid 19937
I20240502 14:13:21.055361 19835 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_zeJNl/build/asan/bin/kudu pid 20068
I20240502 14:13:21.056484 20190 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:13:21.143487 19835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_zeJNl/build/asan/bin/kudu with pid 20068
I20240502 14:13:21.161564 19835 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_zeJNl/build/asan/bin/kudu pid 20198
I20240502 14:13:21.162668 20320 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:13:21.255226 19835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_zeJNl/build/asan/bin/kudu with pid 20198
I20240502 14:13:21.273561 19835 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_zeJNl/build/asan/bin/kudu pid 20329
I20240502 14:13:21.274555 20450 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:13:21.366652 19835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_zeJNl/build/asan/bin/kudu with pid 20329
I20240502 14:13:21.384709 19835 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_zeJNl/build/asan/bin/kudu pid 20460
I20240502 14:13:21.385810 20582 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:13:21.473896 19835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_zeJNl/build/asan/bin/kudu with pid 20460
I20240502 14:13:21.491663 19835 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_zeJNl/build/asan/bin/kudu pid 19847
I20240502 14:13:21.492877 19908 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:13:21.603050 19835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_zeJNl/build/asan/bin/kudu with pid 19847
2024-05-02T14:13:21Z chronyd exiting
[ OK ] AdminCliTest.RebalancerReportOnly (5648 ms)
[----------] 1 test from AdminCliTest (5648 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.TabletServerIsDown/0
I20240502 14:13:21.647487 19835 test_util.cc:268] Using random seed: 882901843
I20240502 14:13:21.649361 19835 ts_itest-base.cc:112] Starting cluster with:
I20240502 14:13:21.649446 19835 ts_itest-base.cc:113] --------------
I20240502 14:13:21.649492 19835 ts_itest-base.cc:114] 5 tablet servers
I20240502 14:13:21.649550 19835 ts_itest-base.cc:115] 3 replicas per TS
I20240502 14:13:21.649590 19835 ts_itest-base.cc:116] --------------
2024-05-02T14:13:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:13:21Z Disabled control of system clock
I20240502 14:13:21.674433 19835 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_zeJNl/build/asan/bin/kudu
/tmp/dist-test-task_zeJNl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.94.254:45685
--webserver_interface=127.19.94.254
--webserver_port=0
--builtin_ntp_servers=127.19.94.212:45051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.94.254:45685
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240502 14:13:22.022230 20718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:22.022585 20718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:22.022683 20718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:22.030503 20718 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:13:22.030747 20718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:13:22.030813 20718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:22.030869 20718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:13:22.030920 20718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:13:22.041388 20718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.94.212:45051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.19.94.254:45685
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.94.254:45685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.19.94.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:22.042932 20718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:22.045063 20718 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:22.056283 20724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:22.056943 20726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:22.057943 20723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:22.058148 20718 server_base.cc:1034] running on GCE node
I20240502 14:13:22.058990 20718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:22.060510 20718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:22.061731 20718 hybrid_clock.cc:648] HybridClock initialized: now 1714659202061674 us; error 49 us; skew 500 ppm
I20240502 14:13:22.062121 20718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:22.064657 20718 webserver.cc:460] Webserver started at http://127.19.94.254:41295/ using document root <none> and password file <none>
I20240502 14:13:22.065404 20718 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:22.065513 20718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:22.065819 20718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:22.068583 20718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "be6bb1797a5d4d958ee56dd401b12389"
format_stamp: "Formatted at 2024-05-02 14:13:22 on dist-test-slave-3j2v"
I20240502 14:13:22.069346 20718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "be6bb1797a5d4d958ee56dd401b12389"
format_stamp: "Formatted at 2024-05-02 14:13:22 on dist-test-slave-3j2v"
I20240502 14:13:22.075018 20718 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 14:13:22.078743 20732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:22.080178 20718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 14:13:22.080376 20718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/wal
uuid: "be6bb1797a5d4d958ee56dd401b12389"
format_stamp: "Formatted at 2024-05-02 14:13:22 on dist-test-slave-3j2v"
I20240502 14:13:22.080657 20718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:22.104518 20718 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:22.105646 20718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:22.106071 20718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:22.125237 20718 rpc_server.cc:307] RPC server started. Bound to: 127.19.94.254:45685
I20240502 14:13:22.125262 20783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.94.254:45685 every 8 connection(s)
I20240502 14:13:22.126662 20718 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 14:13:22.130501 20784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:22.134019 19835 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_zeJNl/build/asan/bin/kudu as pid 20718
I20240502 14:13:22.134253 19835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 14:13:22.142522 20784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be6bb1797a5d4d958ee56dd401b12389: Bootstrap starting.
I20240502 14:13:22.146526 20784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be6bb1797a5d4d958ee56dd401b12389: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:22.148036 20784 log.cc:824] T 00000000000000000000000000000000 P be6bb1797a5d4d958ee56dd401b12389: Log is configured to *not* fsync() on all Append() calls
I20240502 14:13:22.151032 20784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be6bb1797a5d4d958ee56dd401b12389: No bootstrap required, opened a new log
I20240502 14:13:22.157920 20784 raft_consensus.cc:348] T 00000000000000000000000000000000 P be6bb1797a5d4d958ee56dd401b12389 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be6bb1797a5d4d958ee56dd401b12389" member_type: VOTER last_known_addr { host: "127.19.94.254" port: 45685 } }
I20240502 14:13:22.158320 20784 raft_consensus.cc:374] T 00000000000000000000000000000000 P be6bb1797a5d4d958ee56dd401b12389 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:22.158444 20784 raft_consensus.cc:729] T 00000000000000000000000000000000 P be6bb1797a5d4d958ee56dd401b12389 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be6bb1797a5d4d958ee56dd401b12389, State: Initialized, Role: FOLLOWER
I20240502 14:13:22.159137 20784 consensus_queue.cc:260] T 00000000000000000000000000000000 P be6bb1797a5d4d958ee56dd401b12389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be6bb1797a5d4d958ee56dd401b12389" member_type: VOTER last_known_addr { host: "127.19.94.254" port: 45685 } }
I20240502 14:13:22.159345 20784 raft_consensus.cc:388] T 00000000000000000000000000000000 P be6bb1797a5d4d958ee56dd401b12389 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:22.159478 20784 raft_consensus.cc:482] T 00000000000000000000000000000000 P be6bb1797a5d4d958ee56dd401b12389 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:22.159648 20784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be6bb1797a5d4d958ee56dd401b12389 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:22.162253 20784 raft_consensus.cc:504] T 00000000000000000000000000000000 P be6bb1797a5d4d958ee56dd401b12389 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be6bb1797a5d4d958ee56dd401b12389" member_type: VOTER last_known_addr { host: "127.19.94.254" port: 45685 } }
I20240502 14:13:22.162787 20784 leader_election.cc:302] T 00000000000000000000000000000000 P be6bb1797a5d4d958ee56dd401b12389 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be6bb1797a5d4d958ee56dd401b12389; no voters:
I20240502 14:13:22.163421 20784 leader_election.cc:288] T 00000000000000000000000000000000 P be6bb1797a5d4d958ee56dd401b12389 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:22.163599 20789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be6bb1797a5d4d958ee56dd401b12389 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:22.164366 20789 raft_consensus.cc:686] T 00000000000000000000000000000000 P be6bb1797a5d4d958ee56dd401b12389 [term 1 LEADER]: Becoming Leader. State: Replica: be6bb1797a5d4d958ee56dd401b12389, State: Running, Role: LEADER
I20240502 14:13:22.164974 20789 consensus_queue.cc:237] T 00000000000000000000000000000000 P be6bb1797a5d4d958ee56dd401b12389 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be6bb1797a5d4d958ee56dd401b12389" member_type: VOTER last_known_addr { host: "127.19.94.254" port: 45685 } }
I20240502 14:13:22.165656 20784 sys_catalog.cc:564] T 00000000000000000000000000000000 P be6bb1797a5d4d958ee56dd401b12389 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:13:22.167904 20791 sys_catalog.cc:455] T 00000000000000000000000000000000 P be6bb1797a5d4d958ee56dd401b12389 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be6bb1797a5d4d958ee56dd401b12389. Latest consensus state: current_term: 1 leader_uuid: "be6bb1797a5d4d958ee56dd401b12389" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be6bb1797a5d4d958ee56dd401b12389" member_type: VOTER last_known_addr { host: "127.19.94.254" port: 45685 } } }
I20240502 14:13:22.168339 20791 sys_catalog.cc:458] T 00000000000000000000000000000000 P be6bb1797a5d4d958ee56dd401b12389 [sys.catalog]: This master's current role is: LEADER
I20240502 14:13:22.169073 20790 sys_catalog.cc:455] T 00000000000000000000000000000000 P be6bb1797a5d4d958ee56dd401b12389 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be6bb1797a5d4d958ee56dd401b12389" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be6bb1797a5d4d958ee56dd401b12389" member_type: VOTER last_known_addr { host: "127.19.94.254" port: 45685 } } }
I20240502 14:13:22.169379 20790 sys_catalog.cc:458] T 00000000000000000000000000000000 P be6bb1797a5d4d958ee56dd401b12389 [sys.catalog]: This master's current role is: LEADER
I20240502 14:13:22.170753 20796 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:13:22.175837 20796 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:13:22.184496 20796 catalog_manager.cc:1347] Generated new cluster ID: 113e3d389180436c807d33bac9becf48
I20240502 14:13:22.184656 20796 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:13:22.202517 20796 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:13:22.204425 20796 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:13:22.229615 20796 catalog_manager.cc:5857] T 00000000000000000000000000000000 P be6bb1797a5d4d958ee56dd401b12389: Generated new TSK 0
I20240502 14:13:22.230804 20796 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:13:22.248032 19835 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_zeJNl/build/asan/bin/kudu
/tmp/dist-test-task_zeJNl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.94.193:0
--local_ip_for_outbound_sockets=127.19.94.193
--webserver_interface=127.19.94.193
--webserver_port=0
--tserver_master_addrs=127.19.94.254:45685
--builtin_ntp_servers=127.19.94.212:45051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 14:13:22.589681 20808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:22.589980 20808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:22.590103 20808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:22.597759 20808 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:13:22.597891 20808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:22.598057 20808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.94.193
I20240502 14:13:22.607697 20808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.94.212:45051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.94.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.19.94.193
--webserver_port=0
--tserver_master_addrs=127.19.94.254:45685
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.94.193
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:22.609131 20808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:22.611083 20808 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:22.624178 20814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:22.624205 20813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:22.625069 20816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:22.625922 20808 server_base.cc:1034] running on GCE node
I20240502 14:13:22.626613 20808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:22.627673 20808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:22.628901 20808 hybrid_clock.cc:648] HybridClock initialized: now 1714659202628854 us; error 33 us; skew 500 ppm
I20240502 14:13:22.629254 20808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:22.631733 20808 webserver.cc:460] Webserver started at http://127.19.94.193:41475/ using document root <none> and password file <none>
I20240502 14:13:22.632596 20808 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:22.632705 20808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:22.633044 20808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:22.635684 20808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "b52f27892f8142bdaa476ac0b7ef05c1"
format_stamp: "Formatted at 2024-05-02 14:13:22 on dist-test-slave-3j2v"
I20240502 14:13:22.636546 20808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "b52f27892f8142bdaa476ac0b7ef05c1"
format_stamp: "Formatted at 2024-05-02 14:13:22 on dist-test-slave-3j2v"
I20240502 14:13:22.642055 20808 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20240502 14:13:22.645726 20822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:22.647287 20808 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 14:13:22.647490 20808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b52f27892f8142bdaa476ac0b7ef05c1"
format_stamp: "Formatted at 2024-05-02 14:13:22 on dist-test-slave-3j2v"
I20240502 14:13:22.647717 20808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:22.670029 20808 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:22.670982 20808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:22.671257 20808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:22.672276 20808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:22.673885 20808 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:22.673988 20808 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:22.674113 20808 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:22.674190 20808 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:22.707131 20808 rpc_server.cc:307] RPC server started. Bound to: 127.19.94.193:44489
I20240502 14:13:22.707183 20934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.94.193:44489 every 8 connection(s)
I20240502 14:13:22.708563 20808 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 14:13:22.709460 19835 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_zeJNl/build/asan/bin/kudu as pid 20808
I20240502 14:13:22.709698 19835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 14:13:22.713750 19835 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_zeJNl/build/asan/bin/kudu
/tmp/dist-test-task_zeJNl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.94.194:0
--local_ip_for_outbound_sockets=127.19.94.194
--webserver_interface=127.19.94.194
--webserver_port=0
--tserver_master_addrs=127.19.94.254:45685
--builtin_ntp_servers=127.19.94.212:45051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:13:22.722955 20935 heartbeater.cc:347] Connected to a master server at 127.19.94.254:45685
I20240502 14:13:22.723322 20935 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:22.724179 20935 heartbeater.cc:511] Master 127.19.94.254:45685 requested a full tablet report, sending...
I20240502 14:13:22.726553 20749 ts_manager.cc:194] Registered new tserver with Master: b52f27892f8142bdaa476ac0b7ef05c1 (127.19.94.193:44489)
I20240502 14:13:22.728297 20749 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.94.193:38843
W20240502 14:13:23.052160 20939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:23.052484 20939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:23.052636 20939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:23.060848 20939 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:13:23.060983 20939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:23.061141 20939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.94.194
I20240502 14:13:23.070452 20939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.94.212:45051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.94.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.19.94.194
--webserver_port=0
--tserver_master_addrs=127.19.94.254:45685
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.94.194
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:23.071861 20939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:23.073678 20939 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:23.085516 20947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:23.086181 20945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:23.086894 20944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:23.087628 20939 server_base.cc:1034] running on GCE node
I20240502 14:13:23.088294 20939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:23.089365 20939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:23.090571 20939 hybrid_clock.cc:648] HybridClock initialized: now 1714659203090528 us; error 48 us; skew 500 ppm
I20240502 14:13:23.090893 20939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:23.093340 20939 webserver.cc:460] Webserver started at http://127.19.94.194:35565/ using document root <none> and password file <none>
I20240502 14:13:23.094099 20939 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:23.094221 20939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:23.094543 20939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:23.097241 20939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "9b53842cb149443f84f205fe6ad8d941"
format_stamp: "Formatted at 2024-05-02 14:13:23 on dist-test-slave-3j2v"
I20240502 14:13:23.098013 20939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "9b53842cb149443f84f205fe6ad8d941"
format_stamp: "Formatted at 2024-05-02 14:13:23 on dist-test-slave-3j2v"
I20240502 14:13:23.103263 20939 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240502 14:13:23.106871 20953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:23.108328 20939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240502 14:13:23.108531 20939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "9b53842cb149443f84f205fe6ad8d941"
format_stamp: "Formatted at 2024-05-02 14:13:23 on dist-test-slave-3j2v"
I20240502 14:13:23.108744 20939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:23.123229 20939 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:23.124156 20939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:23.124511 20939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:23.125633 20939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:23.127224 20939 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:23.127326 20939 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:23.127451 20939 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:23.127525 20939 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:23.163060 20939 rpc_server.cc:307] RPC server started. Bound to: 127.19.94.194:46013
I20240502 14:13:23.163106 21065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.94.194:46013 every 8 connection(s)
I20240502 14:13:23.164624 20939 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 14:13:23.173218 19835 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_zeJNl/build/asan/bin/kudu as pid 20939
I20240502 14:13:23.173411 19835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 14:13:23.176913 19835 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_zeJNl/build/asan/bin/kudu
/tmp/dist-test-task_zeJNl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.94.195:0
--local_ip_for_outbound_sockets=127.19.94.195
--webserver_interface=127.19.94.195
--webserver_port=0
--tserver_master_addrs=127.19.94.254:45685
--builtin_ntp_servers=127.19.94.212:45051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:13:23.177728 21066 heartbeater.cc:347] Connected to a master server at 127.19.94.254:45685
I20240502 14:13:23.178047 21066 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:23.178860 21066 heartbeater.cc:511] Master 127.19.94.254:45685 requested a full tablet report, sending...
I20240502 14:13:23.180366 20749 ts_manager.cc:194] Registered new tserver with Master: 9b53842cb149443f84f205fe6ad8d941 (127.19.94.194:46013)
I20240502 14:13:23.181345 20749 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.94.194:44453
W20240502 14:13:23.513909 21069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:23.514233 21069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:23.514323 21069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:23.521843 21069 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:13:23.521981 21069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:23.522099 21069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.94.195
I20240502 14:13:23.531170 21069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.94.212:45051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.94.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.19.94.195
--webserver_port=0
--tserver_master_addrs=127.19.94.254:45685
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.94.195
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:23.532472 21069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:23.534235 21069 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:23.546156 21074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:23.546792 21075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:23.547542 21077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:23.548287 21069 server_base.cc:1034] running on GCE node
I20240502 14:13:23.548946 21069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:23.549858 21069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:23.551056 21069 hybrid_clock.cc:648] HybridClock initialized: now 1714659203551009 us; error 34 us; skew 500 ppm
I20240502 14:13:23.551373 21069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:23.553777 21069 webserver.cc:460] Webserver started at http://127.19.94.195:39157/ using document root <none> and password file <none>
I20240502 14:13:23.554528 21069 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:23.554658 21069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:23.555073 21069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:23.558523 21069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "cdcdd4addf624018942b81fc083a7433"
format_stamp: "Formatted at 2024-05-02 14:13:23 on dist-test-slave-3j2v"
I20240502 14:13:23.559310 21069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "cdcdd4addf624018942b81fc083a7433"
format_stamp: "Formatted at 2024-05-02 14:13:23 on dist-test-slave-3j2v"
I20240502 14:13:23.564672 21069 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20240502 14:13:23.568215 21083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:23.569556 21069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:13:23.569730 21069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "cdcdd4addf624018942b81fc083a7433"
format_stamp: "Formatted at 2024-05-02 14:13:23 on dist-test-slave-3j2v"
I20240502 14:13:23.569924 21069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:23.590009 21069 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:23.590965 21069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:23.591245 21069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:23.592286 21069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:23.593851 21069 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:23.593951 21069 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:23.594071 21069 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:23.594146 21069 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:23.627456 21069 rpc_server.cc:307] RPC server started. Bound to: 127.19.94.195:44093
I20240502 14:13:23.627528 21195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.94.195:44093 every 8 connection(s)
I20240502 14:13:23.628899 21069 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 14:13:23.637379 19835 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_zeJNl/build/asan/bin/kudu as pid 21069
I20240502 14:13:23.637562 19835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 14:13:23.641196 19835 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_zeJNl/build/asan/bin/kudu
/tmp/dist-test-task_zeJNl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.94.196:0
--local_ip_for_outbound_sockets=127.19.94.196
--webserver_interface=127.19.94.196
--webserver_port=0
--tserver_master_addrs=127.19.94.254:45685
--builtin_ntp_servers=127.19.94.212:45051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:13:23.642805 21196 heartbeater.cc:347] Connected to a master server at 127.19.94.254:45685
I20240502 14:13:23.643196 21196 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:23.644256 21196 heartbeater.cc:511] Master 127.19.94.254:45685 requested a full tablet report, sending...
I20240502 14:13:23.646068 20749 ts_manager.cc:194] Registered new tserver with Master: cdcdd4addf624018942b81fc083a7433 (127.19.94.195:44093)
I20240502 14:13:23.647455 20749 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.94.195:50485
I20240502 14:13:23.732194 20935 heartbeater.cc:503] Master 127.19.94.254:45685 was elected leader, sending a full tablet report...
W20240502 14:13:23.990585 21199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:23.990932 21199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:23.991076 21199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:23.998749 21199 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:13:23.998883 21199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:23.999058 21199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.94.196
I20240502 14:13:24.008327 21199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.94.212:45051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.94.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.19.94.196
--webserver_port=0
--tserver_master_addrs=127.19.94.254:45685
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.94.196
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:24.009733 21199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:24.011525 21199 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:24.024356 21204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:24.025012 21205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:24.025700 21207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:24.026441 21199 server_base.cc:1034] running on GCE node
I20240502 14:13:24.027086 21199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:24.028076 21199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:24.029273 21199 hybrid_clock.cc:648] HybridClock initialized: now 1714659204029224 us; error 45 us; skew 500 ppm
I20240502 14:13:24.029594 21199 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:24.031862 21199 webserver.cc:460] Webserver started at http://127.19.94.196:42571/ using document root <none> and password file <none>
I20240502 14:13:24.032683 21199 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:24.032804 21199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:24.033130 21199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:24.035984 21199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "bbf4ffb042ce49a3a275a71a6c6c4766"
format_stamp: "Formatted at 2024-05-02 14:13:24 on dist-test-slave-3j2v"
I20240502 14:13:24.036809 21199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "bbf4ffb042ce49a3a275a71a6c6c4766"
format_stamp: "Formatted at 2024-05-02 14:13:24 on dist-test-slave-3j2v"
I20240502 14:13:24.042174 21199 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240502 14:13:24.045651 21213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:24.046948 21199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 14:13:24.047132 21199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "bbf4ffb042ce49a3a275a71a6c6c4766"
format_stamp: "Formatted at 2024-05-02 14:13:24 on dist-test-slave-3j2v"
I20240502 14:13:24.047317 21199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:24.063486 21199 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:24.064340 21199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:24.064639 21199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:24.065621 21199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:24.067221 21199 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:24.067317 21199 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:24.067446 21199 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:24.067513 21199 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:24.101099 21199 rpc_server.cc:307] RPC server started. Bound to: 127.19.94.196:45521
I20240502 14:13:24.101166 21325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.94.196:45521 every 8 connection(s)
I20240502 14:13:24.102485 21199 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 14:13:24.112167 19835 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_zeJNl/build/asan/bin/kudu as pid 21199
I20240502 14:13:24.112416 19835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 14:13:24.115864 19835 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_zeJNl/build/asan/bin/kudu
/tmp/dist-test-task_zeJNl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.94.197:0
--local_ip_for_outbound_sockets=127.19.94.197
--webserver_interface=127.19.94.197
--webserver_port=0
--tserver_master_addrs=127.19.94.254:45685
--builtin_ntp_servers=127.19.94.212:45051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:13:24.116219 21326 heartbeater.cc:347] Connected to a master server at 127.19.94.254:45685
I20240502 14:13:24.116603 21326 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:24.117597 21326 heartbeater.cc:511] Master 127.19.94.254:45685 requested a full tablet report, sending...
I20240502 14:13:24.119217 20749 ts_manager.cc:194] Registered new tserver with Master: bbf4ffb042ce49a3a275a71a6c6c4766 (127.19.94.196:45521)
I20240502 14:13:24.120015 20749 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.94.196:56681
I20240502 14:13:24.184888 21066 heartbeater.cc:503] Master 127.19.94.254:45685 was elected leader, sending a full tablet report...
W20240502 14:13:24.460851 21329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:24.461154 21329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:24.461256 21329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:24.468910 21329 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:13:24.469030 21329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:24.469161 21329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.94.197
I20240502 14:13:24.478403 21329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.94.212:45051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.94.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.19.94.197
--webserver_port=0
--tserver_master_addrs=127.19.94.254:45685
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.94.197
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:24.479717 21329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:24.481482 21329 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:24.493340 21335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:24.494138 21329 server_base.cc:1034] running on GCE node
W20240502 14:13:24.495141 21334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:24.495194 21337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:24.495929 21329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:24.496932 21329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:24.498160 21329 hybrid_clock.cc:648] HybridClock initialized: now 1714659204498108 us; error 37 us; skew 500 ppm
I20240502 14:13:24.498500 21329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:24.501293 21329 webserver.cc:460] Webserver started at http://127.19.94.197:33521/ using document root <none> and password file <none>
I20240502 14:13:24.502138 21329 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:24.502255 21329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:24.502559 21329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:24.505400 21329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "fa0ff9bde9494bf9bc478f43a94420df"
format_stamp: "Formatted at 2024-05-02 14:13:24 on dist-test-slave-3j2v"
I20240502 14:13:24.506201 21329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "fa0ff9bde9494bf9bc478f43a94420df"
format_stamp: "Formatted at 2024-05-02 14:13:24 on dist-test-slave-3j2v"
I20240502 14:13:24.512248 21329 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20240502 14:13:24.516978 21343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:24.519100 21329 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20240502 14:13:24.519289 21329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "fa0ff9bde9494bf9bc478f43a94420df"
format_stamp: "Formatted at 2024-05-02 14:13:24 on dist-test-slave-3j2v"
I20240502 14:13:24.519505 21329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:24.550813 21329 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:24.551812 21329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:24.552085 21329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:24.553225 21329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:24.554803 21329 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:24.554908 21329 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:24.555033 21329 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:24.555096 21329 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:24.589025 21329 rpc_server.cc:307] RPC server started. Bound to: 127.19.94.197:35625
I20240502 14:13:24.589097 21455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.94.197:35625 every 8 connection(s)
I20240502 14:13:24.590348 21329 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 14:13:24.597044 19835 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_zeJNl/build/asan/bin/kudu as pid 21329
I20240502 14:13:24.597198 19835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 14:13:24.604997 21456 heartbeater.cc:347] Connected to a master server at 127.19.94.254:45685
I20240502 14:13:24.605356 21456 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:24.606289 21456 heartbeater.cc:511] Master 127.19.94.254:45685 requested a full tablet report, sending...
I20240502 14:13:24.608022 20749 ts_manager.cc:194] Registered new tserver with Master: fa0ff9bde9494bf9bc478f43a94420df (127.19.94.197:35625)
I20240502 14:13:24.609002 20749 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.94.197:51299
I20240502 14:13:24.614220 19835 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 14:13:24.631639 20749 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56852:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 14:13:24.650287 21196 heartbeater.cc:503] Master 127.19.94.254:45685 was elected leader, sending a full tablet report...
I20240502 14:13:24.665814 21001 tablet_service.cc:1467] Processing CreateTablet for tablet faf1d04a66494e7b9d25c4835ddb4121 (DEFAULT_TABLE table=TestTable [id=b3b3a910e5b64af49f5c1c85e29f1573]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:13:24.665925 21131 tablet_service.cc:1467] Processing CreateTablet for tablet faf1d04a66494e7b9d25c4835ddb4121 (DEFAULT_TABLE table=TestTable [id=b3b3a910e5b64af49f5c1c85e29f1573]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:13:24.667532 21001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faf1d04a66494e7b9d25c4835ddb4121. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:24.668229 21131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faf1d04a66494e7b9d25c4835ddb4121. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:24.667939 21261 tablet_service.cc:1467] Processing CreateTablet for tablet faf1d04a66494e7b9d25c4835ddb4121 (DEFAULT_TABLE table=TestTable [id=b3b3a910e5b64af49f5c1c85e29f1573]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:13:24.669312 21261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faf1d04a66494e7b9d25c4835ddb4121. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:24.681196 21474 tablet_bootstrap.cc:492] T faf1d04a66494e7b9d25c4835ddb4121 P cdcdd4addf624018942b81fc083a7433: Bootstrap starting.
I20240502 14:13:24.684126 21475 tablet_bootstrap.cc:492] T faf1d04a66494e7b9d25c4835ddb4121 P 9b53842cb149443f84f205fe6ad8d941: Bootstrap starting.
I20240502 14:13:24.686007 21474 tablet_bootstrap.cc:654] T faf1d04a66494e7b9d25c4835ddb4121 P cdcdd4addf624018942b81fc083a7433: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:24.687353 21474 log.cc:824] T faf1d04a66494e7b9d25c4835ddb4121 P cdcdd4addf624018942b81fc083a7433: Log is configured to *not* fsync() on all Append() calls
I20240502 14:13:24.688123 21475 tablet_bootstrap.cc:654] T faf1d04a66494e7b9d25c4835ddb4121 P 9b53842cb149443f84f205fe6ad8d941: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:24.688644 21476 tablet_bootstrap.cc:492] T faf1d04a66494e7b9d25c4835ddb4121 P bbf4ffb042ce49a3a275a71a6c6c4766: Bootstrap starting.
I20240502 14:13:24.689420 21475 log.cc:824] T faf1d04a66494e7b9d25c4835ddb4121 P 9b53842cb149443f84f205fe6ad8d941: Log is configured to *not* fsync() on all Append() calls
I20240502 14:13:24.690282 21474 tablet_bootstrap.cc:492] T faf1d04a66494e7b9d25c4835ddb4121 P cdcdd4addf624018942b81fc083a7433: No bootstrap required, opened a new log
I20240502 14:13:24.690601 21474 ts_tablet_manager.cc:1367] T faf1d04a66494e7b9d25c4835ddb4121 P cdcdd4addf624018942b81fc083a7433: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20240502 14:13:24.692265 21476 tablet_bootstrap.cc:654] T faf1d04a66494e7b9d25c4835ddb4121 P bbf4ffb042ce49a3a275a71a6c6c4766: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:24.693483 21475 tablet_bootstrap.cc:492] T faf1d04a66494e7b9d25c4835ddb4121 P 9b53842cb149443f84f205fe6ad8d941: No bootstrap required, opened a new log
I20240502 14:13:24.693574 21476 log.cc:824] T faf1d04a66494e7b9d25c4835ddb4121 P bbf4ffb042ce49a3a275a71a6c6c4766: Log is configured to *not* fsync() on all Append() calls
I20240502 14:13:24.693781 21475 ts_tablet_manager.cc:1367] T faf1d04a66494e7b9d25c4835ddb4121 P 9b53842cb149443f84f205fe6ad8d941: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20240502 14:13:24.696440 21476 tablet_bootstrap.cc:492] T faf1d04a66494e7b9d25c4835ddb4121 P bbf4ffb042ce49a3a275a71a6c6c4766: No bootstrap required, opened a new log
I20240502 14:13:24.696719 21476 ts_tablet_manager.cc:1367] T faf1d04a66494e7b9d25c4835ddb4121 P bbf4ffb042ce49a3a275a71a6c6c4766: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.002s
I20240502 14:13:24.697563 21474 raft_consensus.cc:348] T faf1d04a66494e7b9d25c4835ddb4121 P cdcdd4addf624018942b81fc083a7433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdcdd4addf624018942b81fc083a7433" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44093 } } peers { permanent_uuid: "bbf4ffb042ce49a3a275a71a6c6c4766" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 45521 } } peers { permanent_uuid: "9b53842cb149443f84f205fe6ad8d941" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 46013 } }
I20240502 14:13:24.697918 21474 raft_consensus.cc:374] T faf1d04a66494e7b9d25c4835ddb4121 P cdcdd4addf624018942b81fc083a7433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:24.698016 21474 raft_consensus.cc:729] T faf1d04a66494e7b9d25c4835ddb4121 P cdcdd4addf624018942b81fc083a7433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdcdd4addf624018942b81fc083a7433, State: Initialized, Role: FOLLOWER
I20240502 14:13:24.698702 21474 consensus_queue.cc:260] T faf1d04a66494e7b9d25c4835ddb4121 P cdcdd4addf624018942b81fc083a7433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdcdd4addf624018942b81fc083a7433" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44093 } } peers { permanent_uuid: "bbf4ffb042ce49a3a275a71a6c6c4766" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 45521 } } peers { permanent_uuid: "9b53842cb149443f84f205fe6ad8d941" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 46013 } }
I20240502 14:13:24.702076 21474 ts_tablet_manager.cc:1398] T faf1d04a66494e7b9d25c4835ddb4121 P cdcdd4addf624018942b81fc083a7433: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20240502 14:13:24.702904 21475 raft_consensus.cc:348] T faf1d04a66494e7b9d25c4835ddb4121 P 9b53842cb149443f84f205fe6ad8d941 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdcdd4addf624018942b81fc083a7433" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44093 } } peers { permanent_uuid: "bbf4ffb042ce49a3a275a71a6c6c4766" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 45521 } } peers { permanent_uuid: "9b53842cb149443f84f205fe6ad8d941" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 46013 } }
I20240502 14:13:24.703433 21475 raft_consensus.cc:374] T faf1d04a66494e7b9d25c4835ddb4121 P 9b53842cb149443f84f205fe6ad8d941 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:24.703549 21475 raft_consensus.cc:729] T faf1d04a66494e7b9d25c4835ddb4121 P 9b53842cb149443f84f205fe6ad8d941 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b53842cb149443f84f205fe6ad8d941, State: Initialized, Role: FOLLOWER
I20240502 14:13:24.704349 21475 consensus_queue.cc:260] T faf1d04a66494e7b9d25c4835ddb4121 P 9b53842cb149443f84f205fe6ad8d941 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdcdd4addf624018942b81fc083a7433" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44093 } } peers { permanent_uuid: "bbf4ffb042ce49a3a275a71a6c6c4766" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 45521 } } peers { permanent_uuid: "9b53842cb149443f84f205fe6ad8d941" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 46013 } }
I20240502 14:13:24.705870 21476 raft_consensus.cc:348] T faf1d04a66494e7b9d25c4835ddb4121 P bbf4ffb042ce49a3a275a71a6c6c4766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdcdd4addf624018942b81fc083a7433" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44093 } } peers { permanent_uuid: "bbf4ffb042ce49a3a275a71a6c6c4766" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 45521 } } peers { permanent_uuid: "9b53842cb149443f84f205fe6ad8d941" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 46013 } }
I20240502 14:13:24.706375 21476 raft_consensus.cc:374] T faf1d04a66494e7b9d25c4835ddb4121 P bbf4ffb042ce49a3a275a71a6c6c4766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:24.706387 21475 ts_tablet_manager.cc:1398] T faf1d04a66494e7b9d25c4835ddb4121 P 9b53842cb149443f84f205fe6ad8d941: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20240502 14:13:24.706496 21476 raft_consensus.cc:729] T faf1d04a66494e7b9d25c4835ddb4121 P bbf4ffb042ce49a3a275a71a6c6c4766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf4ffb042ce49a3a275a71a6c6c4766, State: Initialized, Role: FOLLOWER
I20240502 14:13:24.707281 21476 consensus_queue.cc:260] T faf1d04a66494e7b9d25c4835ddb4121 P bbf4ffb042ce49a3a275a71a6c6c4766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdcdd4addf624018942b81fc083a7433" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44093 } } peers { permanent_uuid: "bbf4ffb042ce49a3a275a71a6c6c4766" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 45521 } } peers { permanent_uuid: "9b53842cb149443f84f205fe6ad8d941" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 46013 } }
I20240502 14:13:24.712839 21326 heartbeater.cc:503] Master 127.19.94.254:45685 was elected leader, sending a full tablet report...
I20240502 14:13:24.713378 21476 ts_tablet_manager.cc:1398] T faf1d04a66494e7b9d25c4835ddb4121 P bbf4ffb042ce49a3a275a71a6c6c4766: Time spent starting tablet: real 0.017s user 0.011s sys 0.005s
I20240502 14:13:24.724974 21480 raft_consensus.cc:482] T faf1d04a66494e7b9d25c4835ddb4121 P cdcdd4addf624018942b81fc083a7433 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:13:24.725263 21480 raft_consensus.cc:504] T faf1d04a66494e7b9d25c4835ddb4121 P cdcdd4addf624018942b81fc083a7433 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdcdd4addf624018942b81fc083a7433" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44093 } } peers { permanent_uuid: "bbf4ffb042ce49a3a275a71a6c6c4766" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 45521 } } peers { permanent_uuid: "9b53842cb149443f84f205fe6ad8d941" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 46013 } }
I20240502 14:13:24.727015 21480 leader_election.cc:288] T faf1d04a66494e7b9d25c4835ddb4121 P cdcdd4addf624018942b81fc083a7433 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf4ffb042ce49a3a275a71a6c6c4766 (127.19.94.196:45521), 9b53842cb149443f84f205fe6ad8d941 (127.19.94.194:46013)
I20240502 14:13:24.736371 21021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf1d04a66494e7b9d25c4835ddb4121" candidate_uuid: "cdcdd4addf624018942b81fc083a7433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b53842cb149443f84f205fe6ad8d941" is_pre_election: true
I20240502 14:13:24.736454 21281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf1d04a66494e7b9d25c4835ddb4121" candidate_uuid: "cdcdd4addf624018942b81fc083a7433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf4ffb042ce49a3a275a71a6c6c4766" is_pre_election: true
I20240502 14:13:24.736945 21021 raft_consensus.cc:2454] T faf1d04a66494e7b9d25c4835ddb4121 P 9b53842cb149443f84f205fe6ad8d941 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdcdd4addf624018942b81fc083a7433 in term 0.
I20240502 14:13:24.737145 21281 raft_consensus.cc:2454] T faf1d04a66494e7b9d25c4835ddb4121 P bbf4ffb042ce49a3a275a71a6c6c4766 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdcdd4addf624018942b81fc083a7433 in term 0.
I20240502 14:13:24.737617 21087 leader_election.cc:302] T faf1d04a66494e7b9d25c4835ddb4121 P cdcdd4addf624018942b81fc083a7433 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b53842cb149443f84f205fe6ad8d941, cdcdd4addf624018942b81fc083a7433; no voters:
I20240502 14:13:24.738056 21480 raft_consensus.cc:2781] T faf1d04a66494e7b9d25c4835ddb4121 P cdcdd4addf624018942b81fc083a7433 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:13:24.738197 21480 raft_consensus.cc:482] T faf1d04a66494e7b9d25c4835ddb4121 P cdcdd4addf624018942b81fc083a7433 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:13:24.738292 21480 raft_consensus.cc:3037] T faf1d04a66494e7b9d25c4835ddb4121 P cdcdd4addf624018942b81fc083a7433 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:24.740475 21480 raft_consensus.cc:504] T faf1d04a66494e7b9d25c4835ddb4121 P cdcdd4addf624018942b81fc083a7433 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdcdd4addf624018942b81fc083a7433" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44093 } } peers { permanent_uuid: "bbf4ffb042ce49a3a275a71a6c6c4766" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 45521 } } peers { permanent_uuid: "9b53842cb149443f84f205fe6ad8d941" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 46013 } }
I20240502 14:13:24.741111 21480 leader_election.cc:288] T faf1d04a66494e7b9d25c4835ddb4121 P cdcdd4addf624018942b81fc083a7433 [CANDIDATE]: Term 1 election: Requested vote from peers bbf4ffb042ce49a3a275a71a6c6c4766 (127.19.94.196:45521), 9b53842cb149443f84f205fe6ad8d941 (127.19.94.194:46013)
I20240502 14:13:24.741573 21281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf1d04a66494e7b9d25c4835ddb4121" candidate_uuid: "cdcdd4addf624018942b81fc083a7433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf4ffb042ce49a3a275a71a6c6c4766"
I20240502 14:13:24.741652 21021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf1d04a66494e7b9d25c4835ddb4121" candidate_uuid: "cdcdd4addf624018942b81fc083a7433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b53842cb149443f84f205fe6ad8d941"
I20240502 14:13:24.741761 21281 raft_consensus.cc:3037] T faf1d04a66494e7b9d25c4835ddb4121 P bbf4ffb042ce49a3a275a71a6c6c4766 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:24.741850 21021 raft_consensus.cc:3037] T faf1d04a66494e7b9d25c4835ddb4121 P 9b53842cb149443f84f205fe6ad8d941 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:24.743858 21281 raft_consensus.cc:2454] T faf1d04a66494e7b9d25c4835ddb4121 P bbf4ffb042ce49a3a275a71a6c6c4766 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdcdd4addf624018942b81fc083a7433 in term 1.
I20240502 14:13:24.743996 21021 raft_consensus.cc:2454] T faf1d04a66494e7b9d25c4835ddb4121 P 9b53842cb149443f84f205fe6ad8d941 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdcdd4addf624018942b81fc083a7433 in term 1.
I20240502 14:13:24.744277 21086 leader_election.cc:302] T faf1d04a66494e7b9d25c4835ddb4121 P cdcdd4addf624018942b81fc083a7433 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbf4ffb042ce49a3a275a71a6c6c4766, cdcdd4addf624018942b81fc083a7433; no voters:
I20240502 14:13:24.744596 21480 raft_consensus.cc:2781] T faf1d04a66494e7b9d25c4835ddb4121 P cdcdd4addf624018942b81fc083a7433 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:24.745076 21480 raft_consensus.cc:686] T faf1d04a66494e7b9d25c4835ddb4121 P cdcdd4addf624018942b81fc083a7433 [term 1 LEADER]: Becoming Leader. State: Replica: cdcdd4addf624018942b81fc083a7433, State: Running, Role: LEADER
I20240502 14:13:24.745601 21480 consensus_queue.cc:237] T faf1d04a66494e7b9d25c4835ddb4121 P cdcdd4addf624018942b81fc083a7433 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdcdd4addf624018942b81fc083a7433" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44093 } } peers { permanent_uuid: "bbf4ffb042ce49a3a275a71a6c6c4766" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 45521 } } peers { permanent_uuid: "9b53842cb149443f84f205fe6ad8d941" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 46013 } }
I20240502 14:13:24.749964 20749 catalog_manager.cc:5484] T faf1d04a66494e7b9d25c4835ddb4121 P cdcdd4addf624018942b81fc083a7433 reported cstate change: term changed from 0 to 1, leader changed from <none> to cdcdd4addf624018942b81fc083a7433 (127.19.94.195). New cstate: current_term: 1 leader_uuid: "cdcdd4addf624018942b81fc083a7433" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdcdd4addf624018942b81fc083a7433" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44093 } } peers { permanent_uuid: "bbf4ffb042ce49a3a275a71a6c6c4766" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 45521 } } peers { permanent_uuid: "9b53842cb149443f84f205fe6ad8d941" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 46013 } } }
I20240502 14:13:24.765028 19835 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 14:13:24.767293 19835 ts_itest-base.cc:233] Waiting for 0 tablets on tserver b52f27892f8142bdaa476ac0b7ef05c1 to finish bootstrapping
I20240502 14:13:24.777253 19835 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 9b53842cb149443f84f205fe6ad8d941 to finish bootstrapping
I20240502 14:13:24.785123 19835 ts_itest-base.cc:233] Waiting for 1 tablets on tserver cdcdd4addf624018942b81fc083a7433 to finish bootstrapping
I20240502 14:13:24.792835 19835 ts_itest-base.cc:233] Waiting for 1 tablets on tserver bbf4ffb042ce49a3a275a71a6c6c4766 to finish bootstrapping
I20240502 14:13:24.800586 19835 ts_itest-base.cc:233] Waiting for 0 tablets on tserver fa0ff9bde9494bf9bc478f43a94420df to finish bootstrapping
I20240502 14:13:24.810791 19835 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_zeJNl/build/asan/bin/kudu pid 20808
I20240502 14:13:24.812006 20930 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 14:13:24.854633 21327 tablet.cc:2369] T faf1d04a66494e7b9d25c4835ddb4121 P bbf4ffb042ce49a3a275a71a6c6c4766: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 14:13:24.881471 21197 tablet.cc:2369] T faf1d04a66494e7b9d25c4835ddb4121 P cdcdd4addf624018942b81fc083a7433: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:13:24.895100 19835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_zeJNl/build/asan/bin/kudu with pid 20808
W20240502 14:13:24.917814 21067 tablet.cc:2369] T faf1d04a66494e7b9d25c4835ddb4121 P 9b53842cb149443f84f205fe6ad8d941: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:13:25.129082 21480 consensus_queue.cc:1035] T faf1d04a66494e7b9d25c4835ddb4121 P cdcdd4addf624018942b81fc083a7433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b53842cb149443f84f205fe6ad8d941" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:13:25.138614 21480 consensus_queue.cc:1035] T faf1d04a66494e7b9d25c4835ddb4121 P cdcdd4addf624018942b81fc083a7433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf4ffb042ce49a3a275a71a6c6c4766" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 45521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:13:25.393548 21261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:25.393641 21001 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:25.393797 21391 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:25.394783 21131 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:25.527740 21001 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:25.536098 21261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:25.546424 21131 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:25.549831 21391 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:25.611793 21456 heartbeater.cc:503] Master 127.19.94.254:45685 was elected leader, sending a full tablet report...
I20240502 14:13:25.709956 21001 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:25.713874 21261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:25.717352 21391 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:25.727188 21131 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:26.349748 21001 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:26.349895 21261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:26.359797 21391 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:26.368680 21131 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:26.503320 21001 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:26.507688 21391 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:26.509496 21261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:26.517338 21131 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:26.674468 21001 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:26.676949 21261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:26.677697 21131 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:26.679111 21391 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:26.832325 19835 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_zeJNl/build/asan/bin/kudu pid 20939
I20240502 14:13:26.834079 21061 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:13:26.933929 19835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_zeJNl/build/asan/bin/kudu with pid 20939
I20240502 14:13:26.953596 19835 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_zeJNl/build/asan/bin/kudu pid 21069
I20240502 14:13:26.954861 21191 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:13:27.062547 19835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_zeJNl/build/asan/bin/kudu with pid 21069
I20240502 14:13:27.083026 19835 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_zeJNl/build/asan/bin/kudu pid 21199
I20240502 14:13:27.084200 21321 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:13:27.185757 19835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_zeJNl/build/asan/bin/kudu with pid 21199
W20240502 14:13:27.204023 20736 connection.cc:535] server connection from 127.19.94.196:56681 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:13:27.204210 19835 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_zeJNl/build/asan/bin/kudu pid 21329
I20240502 14:13:27.205351 21450 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:13:27.298046 19835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_zeJNl/build/asan/bin/kudu with pid 21329
I20240502 14:13:27.319065 19835 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_zeJNl/build/asan/bin/kudu pid 20718
I20240502 14:13:27.320250 20779 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:13:27.444382 19835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_zeJNl/build/asan/bin/kudu with pid 20718
2024-05-02T14:13:27Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.TabletServerIsDown/0 (5842 ms)
[----------] 1 test from RebalanceStartCriteriaTest (5842 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/0
I20240502 14:13:27.490100 19835 test_util.cc:268] Using random seed: 888744465
I20240502 14:13:27.492069 19835 ts_itest-base.cc:112] Starting cluster with:
I20240502 14:13:27.492147 19835 ts_itest-base.cc:113] --------------
I20240502 14:13:27.492225 19835 ts_itest-base.cc:114] 7 tablet servers
I20240502 14:13:27.492282 19835 ts_itest-base.cc:115] 1 replicas per TS
I20240502 14:13:27.492357 19835 ts_itest-base.cc:116] --------------
2024-05-02T14:13:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:13:27Z Disabled control of system clock
I20240502 14:13:27.520279 19835 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_zeJNl/build/asan/bin/kudu
/tmp/dist-test-task_zeJNl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.94.254:40967
--webserver_interface=127.19.94.254
--webserver_port=0
--builtin_ntp_servers=127.19.94.212:39595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.94.254:40967
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240502 14:13:27.871374 21692 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 14:13:27.871762 21692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:27.871874 21692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:27.871981 21692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:27.880506 21692 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:13:27.880661 21692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:13:27.880771 21692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:27.880839 21692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:13:27.880926 21692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:13:27.890663 21692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.94.212:39595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.19.94.254:40967
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.94.254:40967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.19.94.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:27.892095 21692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:27.893958 21692 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:27.906242 21697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:27.906800 21698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:27.907778 21700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:27.908577 21692 server_base.cc:1034] running on GCE node
I20240502 14:13:27.909703 21692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:27.911653 21692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:27.912899 21692 hybrid_clock.cc:648] HybridClock initialized: now 1714659207912825 us; error 44 us; skew 500 ppm
I20240502 14:13:27.913280 21692 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:27.915774 21692 webserver.cc:460] Webserver started at http://127.19.94.254:33777/ using document root <none> and password file <none>
I20240502 14:13:27.917181 21692 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:27.917315 21692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:27.917649 21692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:27.920580 21692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "66ce354883f14c8392e7fc8e9383bc85"
format_stamp: "Formatted at 2024-05-02 14:13:27 on dist-test-slave-3j2v"
I20240502 14:13:27.921414 21692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "66ce354883f14c8392e7fc8e9383bc85"
format_stamp: "Formatted at 2024-05-02 14:13:27 on dist-test-slave-3j2v"
I20240502 14:13:27.927078 21692 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20240502 14:13:27.930625 21706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:27.932080 21692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 14:13:27.932265 21692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/wal
uuid: "66ce354883f14c8392e7fc8e9383bc85"
format_stamp: "Formatted at 2024-05-02 14:13:27 on dist-test-slave-3j2v"
I20240502 14:13:27.932504 21692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:27.948856 21692 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:27.949743 21692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:27.950047 21692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:27.969453 21692 rpc_server.cc:307] RPC server started. Bound to: 127.19.94.254:40967
I20240502 14:13:27.969476 21757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.94.254:40967 every 8 connection(s)
I20240502 14:13:27.971033 21692 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 14:13:27.975111 21758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:27.981420 19835 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_zeJNl/build/asan/bin/kudu as pid 21692
I20240502 14:13:27.981685 19835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 14:13:27.986162 21758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66ce354883f14c8392e7fc8e9383bc85: Bootstrap starting.
I20240502 14:13:27.989697 21758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66ce354883f14c8392e7fc8e9383bc85: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:27.990983 21758 log.cc:824] T 00000000000000000000000000000000 P 66ce354883f14c8392e7fc8e9383bc85: Log is configured to *not* fsync() on all Append() calls
I20240502 14:13:27.993986 21758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66ce354883f14c8392e7fc8e9383bc85: No bootstrap required, opened a new log
I20240502 14:13:28.001117 21758 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66ce354883f14c8392e7fc8e9383bc85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ce354883f14c8392e7fc8e9383bc85" member_type: VOTER last_known_addr { host: "127.19.94.254" port: 40967 } }
I20240502 14:13:28.001490 21758 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66ce354883f14c8392e7fc8e9383bc85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:28.001613 21758 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66ce354883f14c8392e7fc8e9383bc85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66ce354883f14c8392e7fc8e9383bc85, State: Initialized, Role: FOLLOWER
I20240502 14:13:28.002286 21758 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66ce354883f14c8392e7fc8e9383bc85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ce354883f14c8392e7fc8e9383bc85" member_type: VOTER last_known_addr { host: "127.19.94.254" port: 40967 } }
I20240502 14:13:28.002549 21758 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66ce354883f14c8392e7fc8e9383bc85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:28.002693 21758 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66ce354883f14c8392e7fc8e9383bc85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:28.002897 21758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66ce354883f14c8392e7fc8e9383bc85 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:28.008720 21758 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66ce354883f14c8392e7fc8e9383bc85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ce354883f14c8392e7fc8e9383bc85" member_type: VOTER last_known_addr { host: "127.19.94.254" port: 40967 } }
I20240502 14:13:28.009485 21758 leader_election.cc:302] T 00000000000000000000000000000000 P 66ce354883f14c8392e7fc8e9383bc85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66ce354883f14c8392e7fc8e9383bc85; no voters:
I20240502 14:13:28.010203 21758 leader_election.cc:288] T 00000000000000000000000000000000 P 66ce354883f14c8392e7fc8e9383bc85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:28.010383 21763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66ce354883f14c8392e7fc8e9383bc85 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:28.011710 21763 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66ce354883f14c8392e7fc8e9383bc85 [term 1 LEADER]: Becoming Leader. State: Replica: 66ce354883f14c8392e7fc8e9383bc85, State: Running, Role: LEADER
I20240502 14:13:28.012452 21763 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66ce354883f14c8392e7fc8e9383bc85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ce354883f14c8392e7fc8e9383bc85" member_type: VOTER last_known_addr { host: "127.19.94.254" port: 40967 } }
I20240502 14:13:28.012667 21758 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66ce354883f14c8392e7fc8e9383bc85 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:13:28.015506 21764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66ce354883f14c8392e7fc8e9383bc85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66ce354883f14c8392e7fc8e9383bc85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ce354883f14c8392e7fc8e9383bc85" member_type: VOTER last_known_addr { host: "127.19.94.254" port: 40967 } } }
I20240502 14:13:28.015908 21764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66ce354883f14c8392e7fc8e9383bc85 [sys.catalog]: This master's current role is: LEADER
I20240502 14:13:28.015532 21765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66ce354883f14c8392e7fc8e9383bc85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66ce354883f14c8392e7fc8e9383bc85. Latest consensus state: current_term: 1 leader_uuid: "66ce354883f14c8392e7fc8e9383bc85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ce354883f14c8392e7fc8e9383bc85" member_type: VOTER last_known_addr { host: "127.19.94.254" port: 40967 } } }
I20240502 14:13:28.016360 21765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66ce354883f14c8392e7fc8e9383bc85 [sys.catalog]: This master's current role is: LEADER
I20240502 14:13:28.017230 21769 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:13:28.023280 21769 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:13:28.032106 21769 catalog_manager.cc:1347] Generated new cluster ID: a3a43d626e3d4e829db77a64fd7dcd2f
I20240502 14:13:28.032254 21769 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:13:28.052981 21769 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:13:28.054402 21769 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:13:28.070068 21769 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 66ce354883f14c8392e7fc8e9383bc85: Generated new TSK 0
I20240502 14:13:28.071038 21769 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:13:28.079631 19835 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_zeJNl/build/asan/bin/kudu
/tmp/dist-test-task_zeJNl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.94.193:0
--local_ip_for_outbound_sockets=127.19.94.193
--webserver_interface=127.19.94.193
--webserver_port=0
--tserver_master_addrs=127.19.94.254:40967
--builtin_ntp_servers=127.19.94.212:39595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 14:13:28.441784 21782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:28.442107 21782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:28.442230 21782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:28.450081 21782 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:13:28.450219 21782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:28.450385 21782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.94.193
I20240502 14:13:28.460026 21782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.94.212:39595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.94.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.19.94.193
--webserver_port=0
--tserver_master_addrs=127.19.94.254:40967
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.94.193
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:28.461463 21782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:28.463320 21782 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:28.476845 21787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:28.477488 21788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:28.478348 21790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:28.479176 21782 server_base.cc:1034] running on GCE node
I20240502 14:13:28.479844 21782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:28.480926 21782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:28.482241 21782 hybrid_clock.cc:648] HybridClock initialized: now 1714659208482098 us; error 51 us; skew 500 ppm
I20240502 14:13:28.482667 21782 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:28.485565 21782 webserver.cc:460] Webserver started at http://127.19.94.193:36963/ using document root <none> and password file <none>
I20240502 14:13:28.486358 21782 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:28.486476 21782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:28.486816 21782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:28.489660 21782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "0c1d247517e246c2828007ca0a6d5527"
format_stamp: "Formatted at 2024-05-02 14:13:28 on dist-test-slave-3j2v"
I20240502 14:13:28.490630 21782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "0c1d247517e246c2828007ca0a6d5527"
format_stamp: "Formatted at 2024-05-02 14:13:28 on dist-test-slave-3j2v"
I20240502 14:13:28.496681 21782 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20240502 14:13:28.500705 21796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:28.502380 21782 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 14:13:28.502574 21782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "0c1d247517e246c2828007ca0a6d5527"
format_stamp: "Formatted at 2024-05-02 14:13:28 on dist-test-slave-3j2v"
I20240502 14:13:28.502806 21782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:28.525517 21782 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:28.526471 21782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:28.526784 21782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:28.528014 21782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:28.529946 21782 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:28.530054 21782 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:28.530179 21782 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:28.530252 21782 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:28.567306 21782 rpc_server.cc:307] RPC server started. Bound to: 127.19.94.193:38827
I20240502 14:13:28.567353 21908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.94.193:38827 every 8 connection(s)
I20240502 14:13:28.568681 21782 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 14:13:28.571385 19835 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_zeJNl/build/asan/bin/kudu as pid 21782
I20240502 14:13:28.571628 19835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 14:13:28.575874 19835 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_zeJNl/build/asan/bin/kudu
/tmp/dist-test-task_zeJNl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.94.194:0
--local_ip_for_outbound_sockets=127.19.94.194
--webserver_interface=127.19.94.194
--webserver_port=0
--tserver_master_addrs=127.19.94.254:40967
--builtin_ntp_servers=127.19.94.212:39595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:13:28.584733 21909 heartbeater.cc:347] Connected to a master server at 127.19.94.254:40967
I20240502 14:13:28.585090 21909 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:28.585907 21909 heartbeater.cc:511] Master 127.19.94.254:40967 requested a full tablet report, sending...
I20240502 14:13:28.588183 21723 ts_manager.cc:194] Registered new tserver with Master: 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193:38827)
I20240502 14:13:28.589830 21723 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.94.193:39563
W20240502 14:13:28.916720 21913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:28.917065 21913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:28.917212 21913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:28.925166 21913 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:13:28.925329 21913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:28.925500 21913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.94.194
I20240502 14:13:28.935297 21913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.94.212:39595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.94.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.19.94.194
--webserver_port=0
--tserver_master_addrs=127.19.94.254:40967
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.94.194
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:28.936779 21913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:28.938782 21913 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:28.952915 21921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:28.953465 21918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:28.954434 21919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:28.955441 21913 server_base.cc:1034] running on GCE node
I20240502 14:13:28.956256 21913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:28.957381 21913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:28.958618 21913 hybrid_clock.cc:648] HybridClock initialized: now 1714659208958565 us; error 52 us; skew 500 ppm
I20240502 14:13:28.958981 21913 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:28.966459 21913 webserver.cc:460] Webserver started at http://127.19.94.194:37559/ using document root <none> and password file <none>
I20240502 14:13:28.967307 21913 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:28.967442 21913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:28.967792 21913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:28.971576 21913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "61c05b91821b450f9853ce940d18372b"
format_stamp: "Formatted at 2024-05-02 14:13:28 on dist-test-slave-3j2v"
I20240502 14:13:28.972724 21913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "61c05b91821b450f9853ce940d18372b"
format_stamp: "Formatted at 2024-05-02 14:13:28 on dist-test-slave-3j2v"
I20240502 14:13:28.979596 21913 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20240502 14:13:28.984213 21927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:28.985955 21913 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:13:28.986156 21913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "61c05b91821b450f9853ce940d18372b"
format_stamp: "Formatted at 2024-05-02 14:13:28 on dist-test-slave-3j2v"
I20240502 14:13:28.986424 21913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:29.010984 21913 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:29.011886 21913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:29.012161 21913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:29.013322 21913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:29.015025 21913 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:29.015117 21913 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:29.015211 21913 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:29.015266 21913 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:29.054692 21913 rpc_server.cc:307] RPC server started. Bound to: 127.19.94.194:40163
I20240502 14:13:29.054766 22039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.94.194:40163 every 8 connection(s)
I20240502 14:13:29.056051 21913 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 14:13:29.057833 19835 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_zeJNl/build/asan/bin/kudu as pid 21913
I20240502 14:13:29.058048 19835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 14:13:29.061272 19835 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_zeJNl/build/asan/bin/kudu
/tmp/dist-test-task_zeJNl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.94.195:0
--local_ip_for_outbound_sockets=127.19.94.195
--webserver_interface=127.19.94.195
--webserver_port=0
--tserver_master_addrs=127.19.94.254:40967
--builtin_ntp_servers=127.19.94.212:39595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:13:29.069554 22040 heartbeater.cc:347] Connected to a master server at 127.19.94.254:40967
I20240502 14:13:29.069921 22040 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:29.070940 22040 heartbeater.cc:511] Master 127.19.94.254:40967 requested a full tablet report, sending...
I20240502 14:13:29.073552 21723 ts_manager.cc:194] Registered new tserver with Master: 61c05b91821b450f9853ce940d18372b (127.19.94.194:40163)
I20240502 14:13:29.074398 21723 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.94.194:54865
W20240502 14:13:29.425771 22043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:29.426107 22043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:29.426252 22043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:29.434677 22043 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:13:29.434818 22043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:29.434983 22043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.94.195
I20240502 14:13:29.444950 22043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.94.212:39595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.94.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.19.94.195
--webserver_port=0
--tserver_master_addrs=127.19.94.254:40967
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.94.195
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:29.446393 22043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:29.448221 22043 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:29.461639 22048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:29.462493 22049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:29.463241 22051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:29.463999 22043 server_base.cc:1034] running on GCE node
I20240502 14:13:29.464725 22043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:29.465720 22043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:29.466941 22043 hybrid_clock.cc:648] HybridClock initialized: now 1714659209466883 us; error 48 us; skew 500 ppm
I20240502 14:13:29.467268 22043 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:29.469698 22043 webserver.cc:460] Webserver started at http://127.19.94.195:45307/ using document root <none> and password file <none>
I20240502 14:13:29.470490 22043 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:29.470615 22043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:29.470952 22043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:29.473791 22043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "ff15cd84fa6640b8921bbf05dc65612e"
format_stamp: "Formatted at 2024-05-02 14:13:29 on dist-test-slave-3j2v"
I20240502 14:13:29.474596 22043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "ff15cd84fa6640b8921bbf05dc65612e"
format_stamp: "Formatted at 2024-05-02 14:13:29 on dist-test-slave-3j2v"
I20240502 14:13:29.479931 22043 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240502 14:13:29.483712 22057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:29.485153 22043 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240502 14:13:29.485314 22043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ff15cd84fa6640b8921bbf05dc65612e"
format_stamp: "Formatted at 2024-05-02 14:13:29 on dist-test-slave-3j2v"
I20240502 14:13:29.485512 22043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:29.504673 22043 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:29.505605 22043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:29.505896 22043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:29.506987 22043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:29.508841 22043 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:29.508955 22043 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:29.509076 22043 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:29.509213 22043 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:29.545933 22043 rpc_server.cc:307] RPC server started. Bound to: 127.19.94.195:39713
I20240502 14:13:29.545989 22169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.94.195:39713 every 8 connection(s)
I20240502 14:13:29.547323 22043 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 14:13:29.554448 19835 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_zeJNl/build/asan/bin/kudu as pid 22043
I20240502 14:13:29.554690 19835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 14:13:29.558271 19835 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_zeJNl/build/asan/bin/kudu
/tmp/dist-test-task_zeJNl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.94.196:0
--local_ip_for_outbound_sockets=127.19.94.196
--webserver_interface=127.19.94.196
--webserver_port=0
--tserver_master_addrs=127.19.94.254:40967
--builtin_ntp_servers=127.19.94.212:39595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:13:29.562047 22170 heartbeater.cc:347] Connected to a master server at 127.19.94.254:40967
I20240502 14:13:29.562433 22170 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:29.563429 22170 heartbeater.cc:511] Master 127.19.94.254:40967 requested a full tablet report, sending...
I20240502 14:13:29.565239 21723 ts_manager.cc:194] Registered new tserver with Master: ff15cd84fa6640b8921bbf05dc65612e (127.19.94.195:39713)
I20240502 14:13:29.566577 21723 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.94.195:52025
I20240502 14:13:29.593115 21909 heartbeater.cc:503] Master 127.19.94.254:40967 was elected leader, sending a full tablet report...
W20240502 14:13:29.905238 22173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:29.905539 22173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:29.905674 22173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:29.913520 22173 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:13:29.913656 22173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:29.913836 22173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.94.196
I20240502 14:13:29.923337 22173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.94.212:39595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.94.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.19.94.196
--webserver_port=0
--tserver_master_addrs=127.19.94.254:40967
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.94.196
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:29.924778 22173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:29.926591 22173 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:29.940662 22179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:29.941215 22178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:29.942195 22181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:29.943037 22173 server_base.cc:1034] running on GCE node
I20240502 14:13:29.943871 22173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:29.945024 22173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:29.946262 22173 hybrid_clock.cc:648] HybridClock initialized: now 1714659209946205 us; error 54 us; skew 500 ppm
I20240502 14:13:29.946633 22173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:29.949232 22173 webserver.cc:460] Webserver started at http://127.19.94.196:37395/ using document root <none> and password file <none>
I20240502 14:13:29.950058 22173 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:29.950183 22173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:29.950513 22173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:29.953410 22173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "f8fbf60396bb443697ce265de0eb063e"
format_stamp: "Formatted at 2024-05-02 14:13:29 on dist-test-slave-3j2v"
I20240502 14:13:29.954241 22173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "f8fbf60396bb443697ce265de0eb063e"
format_stamp: "Formatted at 2024-05-02 14:13:29 on dist-test-slave-3j2v"
I20240502 14:13:29.959815 22173 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240502 14:13:29.963502 22187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:29.965054 22173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:13:29.965232 22173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "f8fbf60396bb443697ce265de0eb063e"
format_stamp: "Formatted at 2024-05-02 14:13:29 on dist-test-slave-3j2v"
I20240502 14:13:29.965444 22173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:29.979588 22173 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:29.980509 22173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:29.980795 22173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:29.981755 22173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:29.983317 22173 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:29.983420 22173 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:29.983529 22173 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:29.983600 22173 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:30.018285 22173 rpc_server.cc:307] RPC server started. Bound to: 127.19.94.196:41835
I20240502 14:13:30.018349 22299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.94.196:41835 every 8 connection(s)
I20240502 14:13:30.019773 22173 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 14:13:30.021167 19835 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_zeJNl/build/asan/bin/kudu as pid 22173
I20240502 14:13:30.021451 19835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 14:13:30.025333 19835 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_zeJNl/build/asan/bin/kudu
/tmp/dist-test-task_zeJNl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.94.197:0
--local_ip_for_outbound_sockets=127.19.94.197
--webserver_interface=127.19.94.197
--webserver_port=0
--tserver_master_addrs=127.19.94.254:40967
--builtin_ntp_servers=127.19.94.212:39595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:13:30.034570 22300 heartbeater.cc:347] Connected to a master server at 127.19.94.254:40967
I20240502 14:13:30.034965 22300 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:30.036011 22300 heartbeater.cc:511] Master 127.19.94.254:40967 requested a full tablet report, sending...
I20240502 14:13:30.037842 21723 ts_manager.cc:194] Registered new tserver with Master: f8fbf60396bb443697ce265de0eb063e (127.19.94.196:41835)
I20240502 14:13:30.038708 21723 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.94.196:50855
I20240502 14:13:30.077872 22040 heartbeater.cc:503] Master 127.19.94.254:40967 was elected leader, sending a full tablet report...
W20240502 14:13:30.371485 22303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:30.371774 22303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:30.371862 22303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:30.379371 22303 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:13:30.379500 22303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:30.379617 22303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.94.197
I20240502 14:13:30.388882 22303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.94.212:39595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.94.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.19.94.197
--webserver_port=0
--tserver_master_addrs=127.19.94.254:40967
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.94.197
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:30.390239 22303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:30.392038 22303 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:30.404822 22309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:30.405436 22308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:30.406152 22311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:30.406958 22303 server_base.cc:1034] running on GCE node
I20240502 14:13:30.407614 22303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:30.408736 22303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:30.409989 22303 hybrid_clock.cc:648] HybridClock initialized: now 1714659210409917 us; error 65 us; skew 500 ppm
I20240502 14:13:30.410369 22303 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:30.413055 22303 webserver.cc:460] Webserver started at http://127.19.94.197:37515/ using document root <none> and password file <none>
I20240502 14:13:30.413870 22303 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:30.413985 22303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:30.414358 22303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:30.418133 22303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "92646999d5c64d1aae6f165593b7d14d"
format_stamp: "Formatted at 2024-05-02 14:13:30 on dist-test-slave-3j2v"
I20240502 14:13:30.419317 22303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "92646999d5c64d1aae6f165593b7d14d"
format_stamp: "Formatted at 2024-05-02 14:13:30 on dist-test-slave-3j2v"
I20240502 14:13:30.425557 22303 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20240502 14:13:30.429308 22317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:30.430963 22303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 14:13:30.431159 22303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "92646999d5c64d1aae6f165593b7d14d"
format_stamp: "Formatted at 2024-05-02 14:13:30 on dist-test-slave-3j2v"
I20240502 14:13:30.431382 22303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:30.449594 22303 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:30.450495 22303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:30.450788 22303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:30.451794 22303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:30.453457 22303 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:30.453558 22303 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:30.453691 22303 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:30.453766 22303 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:30.485957 22303 rpc_server.cc:307] RPC server started. Bound to: 127.19.94.197:33817
I20240502 14:13:30.485978 22429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.94.197:33817 every 8 connection(s)
I20240502 14:13:30.487305 22303 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 14:13:30.496277 19835 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_zeJNl/build/asan/bin/kudu as pid 22303
I20240502 14:13:30.496497 19835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 14:13:30.499904 19835 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_zeJNl/build/asan/bin/kudu
/tmp/dist-test-task_zeJNl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.94.198:0
--local_ip_for_outbound_sockets=127.19.94.198
--webserver_interface=127.19.94.198
--webserver_port=0
--tserver_master_addrs=127.19.94.254:40967
--builtin_ntp_servers=127.19.94.212:39595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:13:30.500556 22430 heartbeater.cc:347] Connected to a master server at 127.19.94.254:40967
I20240502 14:13:30.500895 22430 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:30.502118 22430 heartbeater.cc:511] Master 127.19.94.254:40967 requested a full tablet report, sending...
I20240502 14:13:30.503674 21723 ts_manager.cc:194] Registered new tserver with Master: 92646999d5c64d1aae6f165593b7d14d (127.19.94.197:33817)
I20240502 14:13:30.504981 21723 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.94.197:58925
I20240502 14:13:30.569913 22170 heartbeater.cc:503] Master 127.19.94.254:40967 was elected leader, sending a full tablet report...
W20240502 14:13:30.849737 22433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:30.850065 22433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:30.850160 22433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:30.858099 22433 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:13:30.858235 22433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:30.858384 22433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.94.198
I20240502 14:13:30.868752 22433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.94.212:39595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.94.198:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.19.94.198
--webserver_port=0
--tserver_master_addrs=127.19.94.254:40967
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.94.198
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:30.870131 22433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:30.871922 22433 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:30.884711 22438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:30.885373 22439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:30.886256 22441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:30.887082 22433 server_base.cc:1034] running on GCE node
I20240502 14:13:30.887735 22433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:30.888900 22433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:30.890122 22433 hybrid_clock.cc:648] HybridClock initialized: now 1714659210890078 us; error 37 us; skew 500 ppm
I20240502 14:13:30.890468 22433 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:30.892933 22433 webserver.cc:460] Webserver started at http://127.19.94.198:44939/ using document root <none> and password file <none>
I20240502 14:13:30.893718 22433 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:30.893831 22433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:30.894132 22433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:30.896919 22433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "d53de47e3dbd4a37bb81b478ae4d1e0e"
format_stamp: "Formatted at 2024-05-02 14:13:30 on dist-test-slave-3j2v"
I20240502 14:13:30.897703 22433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "d53de47e3dbd4a37bb81b478ae4d1e0e"
format_stamp: "Formatted at 2024-05-02 14:13:30 on dist-test-slave-3j2v"
I20240502 14:13:30.903246 22433 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240502 14:13:30.907600 22447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:30.909354 22433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240502 14:13:30.909528 22433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "d53de47e3dbd4a37bb81b478ae4d1e0e"
format_stamp: "Formatted at 2024-05-02 14:13:30 on dist-test-slave-3j2v"
I20240502 14:13:30.912362 22433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:30.930980 22433 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:30.931963 22433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:30.932236 22433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:30.933431 22433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:30.935107 22433 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:30.935221 22433 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:30.935344 22433 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:30.935412 22433 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:30.970886 22433 rpc_server.cc:307] RPC server started. Bound to: 127.19.94.198:41717
I20240502 14:13:30.970968 22559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.94.198:41717 every 8 connection(s)
I20240502 14:13:30.972286 22433 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 14:13:30.981539 19835 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_zeJNl/build/asan/bin/kudu as pid 22433
I20240502 14:13:30.981712 19835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 14:13:30.985008 19835 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_zeJNl/build/asan/bin/kudu
/tmp/dist-test-task_zeJNl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.94.199:0
--local_ip_for_outbound_sockets=127.19.94.199
--webserver_interface=127.19.94.199
--webserver_port=0
--tserver_master_addrs=127.19.94.254:40967
--builtin_ntp_servers=127.19.94.212:39595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:13:30.985146 22560 heartbeater.cc:347] Connected to a master server at 127.19.94.254:40967
I20240502 14:13:30.985430 22560 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:30.986171 22560 heartbeater.cc:511] Master 127.19.94.254:40967 requested a full tablet report, sending...
I20240502 14:13:30.987449 21723 ts_manager.cc:194] Registered new tserver with Master: d53de47e3dbd4a37bb81b478ae4d1e0e (127.19.94.198:41717)
I20240502 14:13:30.988251 21723 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.94.198:60831
I20240502 14:13:31.042254 22300 heartbeater.cc:503] Master 127.19.94.254:40967 was elected leader, sending a full tablet report...
W20240502 14:13:31.343134 22563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:31.343454 22563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:31.343570 22563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:31.353528 22563 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:13:31.353664 22563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:31.353796 22563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.94.199
I20240502 14:13:31.363219 22563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.94.212:39595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.94.199:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.19.94.199
--webserver_port=0
--tserver_master_addrs=127.19.94.254:40967
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.94.199
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:31.364583 22563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:31.366328 22563 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:31.378676 22568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:31.379540 22569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:31.380198 22571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:31.380911 22563 server_base.cc:1034] running on GCE node
I20240502 14:13:31.381508 22563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:31.382437 22563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:31.383652 22563 hybrid_clock.cc:648] HybridClock initialized: now 1714659211383589 us; error 56 us; skew 500 ppm
I20240502 14:13:31.384006 22563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:31.386355 22563 webserver.cc:460] Webserver started at http://127.19.94.199:43897/ using document root <none> and password file <none>
I20240502 14:13:31.387123 22563 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:31.387238 22563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:31.387555 22563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:31.390368 22563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "5ab00105ba8d4412bfb9d009c4e40352"
format_stamp: "Formatted at 2024-05-02 14:13:31 on dist-test-slave-3j2v"
I20240502 14:13:31.391247 22563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "5ab00105ba8d4412bfb9d009c4e40352"
format_stamp: "Formatted at 2024-05-02 14:13:31 on dist-test-slave-3j2v"
I20240502 14:13:31.397010 22563 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240502 14:13:31.400830 22577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:31.402443 22563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20240502 14:13:31.402647 22563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "5ab00105ba8d4412bfb9d009c4e40352"
format_stamp: "Formatted at 2024-05-02 14:13:31 on dist-test-slave-3j2v"
I20240502 14:13:31.402887 22563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:31.420822 22563 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:31.421803 22563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:31.422101 22563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:31.423321 22563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:31.425035 22563 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:31.425141 22563 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:31.425254 22563 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:31.425333 22563 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:31.461088 22563 rpc_server.cc:307] RPC server started. Bound to: 127.19.94.199:44599
I20240502 14:13:31.461146 22689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.94.199:44599 every 8 connection(s)
I20240502 14:13:31.462502 22563 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 14:13:31.464800 19835 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_zeJNl/build/asan/bin/kudu as pid 22563
I20240502 14:13:31.465021 19835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 14:13:31.475564 22690 heartbeater.cc:347] Connected to a master server at 127.19.94.254:40967
I20240502 14:13:31.475874 22690 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:31.476720 22690 heartbeater.cc:511] Master 127.19.94.254:40967 requested a full tablet report, sending...
I20240502 14:13:31.478166 21723 ts_manager.cc:194] Registered new tserver with Master: 5ab00105ba8d4412bfb9d009c4e40352 (127.19.94.199:44599)
I20240502 14:13:31.479068 21723 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.94.199:43837
I20240502 14:13:31.481107 19835 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 14:13:31.508666 22430 heartbeater.cc:503] Master 127.19.94.254:40967 was elected leader, sending a full tablet report...
I20240502 14:13:31.505586 21722 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48804:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 14:13:31.538951 22365 tablet_service.cc:1467] Processing CreateTablet for tablet 41b085facc334a7da0c7dd937275009f (DEFAULT_TABLE table=TestTable [id=5a36b7ae3140445fa870b28b8388210e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:13:31.540308 22365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41b085facc334a7da0c7dd937275009f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:31.551748 22704 tablet_bootstrap.cc:492] T 41b085facc334a7da0c7dd937275009f P 92646999d5c64d1aae6f165593b7d14d: Bootstrap starting.
I20240502 14:13:31.555567 22704 tablet_bootstrap.cc:654] T 41b085facc334a7da0c7dd937275009f P 92646999d5c64d1aae6f165593b7d14d: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:31.556738 22704 log.cc:824] T 41b085facc334a7da0c7dd937275009f P 92646999d5c64d1aae6f165593b7d14d: Log is configured to *not* fsync() on all Append() calls
I20240502 14:13:31.559584 22704 tablet_bootstrap.cc:492] T 41b085facc334a7da0c7dd937275009f P 92646999d5c64d1aae6f165593b7d14d: No bootstrap required, opened a new log
I20240502 14:13:31.559857 22704 ts_tablet_manager.cc:1367] T 41b085facc334a7da0c7dd937275009f P 92646999d5c64d1aae6f165593b7d14d: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240502 14:13:31.567119 22704 raft_consensus.cc:348] T 41b085facc334a7da0c7dd937275009f P 92646999d5c64d1aae6f165593b7d14d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92646999d5c64d1aae6f165593b7d14d" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 33817 } }
I20240502 14:13:31.567785 22704 raft_consensus.cc:374] T 41b085facc334a7da0c7dd937275009f P 92646999d5c64d1aae6f165593b7d14d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:31.567981 22704 raft_consensus.cc:729] T 41b085facc334a7da0c7dd937275009f P 92646999d5c64d1aae6f165593b7d14d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92646999d5c64d1aae6f165593b7d14d, State: Initialized, Role: FOLLOWER
I20240502 14:13:31.569095 22704 consensus_queue.cc:260] T 41b085facc334a7da0c7dd937275009f P 92646999d5c64d1aae6f165593b7d14d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92646999d5c64d1aae6f165593b7d14d" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 33817 } }
I20240502 14:13:31.569491 22704 raft_consensus.cc:388] T 41b085facc334a7da0c7dd937275009f P 92646999d5c64d1aae6f165593b7d14d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:31.569775 22704 raft_consensus.cc:482] T 41b085facc334a7da0c7dd937275009f P 92646999d5c64d1aae6f165593b7d14d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:31.570058 22704 raft_consensus.cc:3037] T 41b085facc334a7da0c7dd937275009f P 92646999d5c64d1aae6f165593b7d14d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:31.573237 22704 raft_consensus.cc:504] T 41b085facc334a7da0c7dd937275009f P 92646999d5c64d1aae6f165593b7d14d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92646999d5c64d1aae6f165593b7d14d" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 33817 } }
I20240502 14:13:31.573881 22704 leader_election.cc:302] T 41b085facc334a7da0c7dd937275009f P 92646999d5c64d1aae6f165593b7d14d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92646999d5c64d1aae6f165593b7d14d; no voters:
I20240502 14:13:31.574491 22704 leader_election.cc:288] T 41b085facc334a7da0c7dd937275009f P 92646999d5c64d1aae6f165593b7d14d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:31.574645 22706 raft_consensus.cc:2781] T 41b085facc334a7da0c7dd937275009f P 92646999d5c64d1aae6f165593b7d14d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:31.575336 22706 raft_consensus.cc:686] T 41b085facc334a7da0c7dd937275009f P 92646999d5c64d1aae6f165593b7d14d [term 1 LEADER]: Becoming Leader. State: Replica: 92646999d5c64d1aae6f165593b7d14d, State: Running, Role: LEADER
I20240502 14:13:31.576305 22704 ts_tablet_manager.cc:1398] T 41b085facc334a7da0c7dd937275009f P 92646999d5c64d1aae6f165593b7d14d: Time spent starting tablet: real 0.016s user 0.012s sys 0.005s
I20240502 14:13:31.576352 22706 consensus_queue.cc:237] T 41b085facc334a7da0c7dd937275009f P 92646999d5c64d1aae6f165593b7d14d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92646999d5c64d1aae6f165593b7d14d" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 33817 } }
I20240502 14:13:31.581672 21722 catalog_manager.cc:5484] T 41b085facc334a7da0c7dd937275009f P 92646999d5c64d1aae6f165593b7d14d reported cstate change: term changed from 0 to 1, leader changed from <none> to 92646999d5c64d1aae6f165593b7d14d (127.19.94.197). New cstate: current_term: 1 leader_uuid: "92646999d5c64d1aae6f165593b7d14d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92646999d5c64d1aae6f165593b7d14d" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 33817 } } }
I20240502 14:13:31.599730 19835 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 14:13:31.602124 19835 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 0c1d247517e246c2828007ca0a6d5527 to finish bootstrapping
I20240502 14:13:31.615510 19835 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 61c05b91821b450f9853ce940d18372b to finish bootstrapping
I20240502 14:13:31.625373 19835 ts_itest-base.cc:233] Waiting for 0 tablets on tserver ff15cd84fa6640b8921bbf05dc65612e to finish bootstrapping
I20240502 14:13:31.635602 19835 ts_itest-base.cc:233] Waiting for 0 tablets on tserver f8fbf60396bb443697ce265de0eb063e to finish bootstrapping
I20240502 14:13:31.645437 19835 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 92646999d5c64d1aae6f165593b7d14d to finish bootstrapping
I20240502 14:13:31.653465 19835 ts_itest-base.cc:233] Waiting for 0 tablets on tserver d53de47e3dbd4a37bb81b478ae4d1e0e to finish bootstrapping
I20240502 14:13:31.662263 19835 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 5ab00105ba8d4412bfb9d009c4e40352 to finish bootstrapping
I20240502 14:13:31.671309 19835 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_zeJNl/build/asan/bin/kudu pid 22043
I20240502 14:13:31.672307 22165 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:13:31.755388 19835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_zeJNl/build/asan/bin/kudu with pid 22043
I20240502 14:13:31.771795 19835 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_zeJNl/build/asan/bin/kudu pid 22173
I20240502 14:13:31.772933 22295 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:13:31.856935 19835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_zeJNl/build/asan/bin/kudu with pid 22173
I20240502 14:13:31.873521 19835 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_zeJNl/build/asan/bin/kudu pid 22303
I20240502 14:13:31.874706 22425 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:13:31.961447 19835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_zeJNl/build/asan/bin/kudu with pid 22303
I20240502 14:13:31.981416 19835 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_zeJNl/build/asan/bin/kudu pid 22433
I20240502 14:13:31.982609 22555 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:13:32.067791 22560 heartbeater.cc:503] Master 127.19.94.254:40967 was elected leader, sending a full tablet report...
I20240502 14:13:32.068604 19835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_zeJNl/build/asan/bin/kudu with pid 22433
W20240502 14:13:32.081586 21707 connection.cc:535] server connection from 127.19.94.198:60831 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:13:32.081909 19835 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_zeJNl/build/asan/bin/kudu pid 22563
I20240502 14:13:32.082973 22685 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:13:32.166188 19835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_zeJNl/build/asan/bin/kudu with pid 22563
I20240502 14:13:35.934664 21723 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48804:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:13:35.954433 21975 tablet_service.cc:1467] Processing CreateTablet for tablet 54ec5887ceee418c9915a1f043978f83 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ea93e5ddb3404119899abcd427b2771e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:13:35.954437 21843 tablet_service.cc:1467] Processing CreateTablet for tablet 8af0e06b069e424a9ace44e06f12a6d3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ea93e5ddb3404119899abcd427b2771e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:13:35.954478 21844 tablet_service.cc:1467] Processing CreateTablet for tablet 8a2d3ba227dd43eb941b7e1b3e8445ce (DEFAULT_TABLE table=rebalance_test_table_0 [id=ea93e5ddb3404119899abcd427b2771e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:13:35.955817 21975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54ec5887ceee418c9915a1f043978f83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:35.955996 21843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8af0e06b069e424a9ace44e06f12a6d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:35.956486 21844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a2d3ba227dd43eb941b7e1b3e8445ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:35.966663 22731 tablet_bootstrap.cc:492] T 54ec5887ceee418c9915a1f043978f83 P 61c05b91821b450f9853ce940d18372b: Bootstrap starting.
I20240502 14:13:35.970067 22732 tablet_bootstrap.cc:492] T 8af0e06b069e424a9ace44e06f12a6d3 P 0c1d247517e246c2828007ca0a6d5527: Bootstrap starting.
I20240502 14:13:35.970207 22731 tablet_bootstrap.cc:654] T 54ec5887ceee418c9915a1f043978f83 P 61c05b91821b450f9853ce940d18372b: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:35.971604 22731 log.cc:824] T 54ec5887ceee418c9915a1f043978f83 P 61c05b91821b450f9853ce940d18372b: Log is configured to *not* fsync() on all Append() calls
I20240502 14:13:35.973863 22732 tablet_bootstrap.cc:654] T 8af0e06b069e424a9ace44e06f12a6d3 P 0c1d247517e246c2828007ca0a6d5527: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:35.974799 22731 tablet_bootstrap.cc:492] T 54ec5887ceee418c9915a1f043978f83 P 61c05b91821b450f9853ce940d18372b: No bootstrap required, opened a new log
I20240502 14:13:35.975095 22731 ts_tablet_manager.cc:1367] T 54ec5887ceee418c9915a1f043978f83 P 61c05b91821b450f9853ce940d18372b: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20240502 14:13:35.975469 22732 log.cc:824] T 8af0e06b069e424a9ace44e06f12a6d3 P 0c1d247517e246c2828007ca0a6d5527: Log is configured to *not* fsync() on all Append() calls
I20240502 14:13:35.978644 22732 tablet_bootstrap.cc:492] T 8af0e06b069e424a9ace44e06f12a6d3 P 0c1d247517e246c2828007ca0a6d5527: No bootstrap required, opened a new log
I20240502 14:13:35.978917 22732 ts_tablet_manager.cc:1367] T 8af0e06b069e424a9ace44e06f12a6d3 P 0c1d247517e246c2828007ca0a6d5527: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240502 14:13:35.984216 22731 raft_consensus.cc:348] T 54ec5887ceee418c9915a1f043978f83 P 61c05b91821b450f9853ce940d18372b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } }
I20240502 14:13:35.984654 22731 raft_consensus.cc:374] T 54ec5887ceee418c9915a1f043978f83 P 61c05b91821b450f9853ce940d18372b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:35.984766 22731 raft_consensus.cc:729] T 54ec5887ceee418c9915a1f043978f83 P 61c05b91821b450f9853ce940d18372b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61c05b91821b450f9853ce940d18372b, State: Initialized, Role: FOLLOWER
I20240502 14:13:35.985504 22731 consensus_queue.cc:260] T 54ec5887ceee418c9915a1f043978f83 P 61c05b91821b450f9853ce940d18372b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } }
I20240502 14:13:35.985747 22731 raft_consensus.cc:388] T 54ec5887ceee418c9915a1f043978f83 P 61c05b91821b450f9853ce940d18372b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:35.985888 22731 raft_consensus.cc:482] T 54ec5887ceee418c9915a1f043978f83 P 61c05b91821b450f9853ce940d18372b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:35.986063 22731 raft_consensus.cc:3037] T 54ec5887ceee418c9915a1f043978f83 P 61c05b91821b450f9853ce940d18372b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:35.988047 22732 raft_consensus.cc:348] T 8af0e06b069e424a9ace44e06f12a6d3 P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } }
I20240502 14:13:35.988454 22732 raft_consensus.cc:374] T 8af0e06b069e424a9ace44e06f12a6d3 P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:35.988557 22732 raft_consensus.cc:729] T 8af0e06b069e424a9ace44e06f12a6d3 P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c1d247517e246c2828007ca0a6d5527, State: Initialized, Role: FOLLOWER
I20240502 14:13:35.989068 22731 raft_consensus.cc:504] T 54ec5887ceee418c9915a1f043978f83 P 61c05b91821b450f9853ce940d18372b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } }
I20240502 14:13:35.989289 22732 consensus_queue.cc:260] T 8af0e06b069e424a9ace44e06f12a6d3 P 0c1d247517e246c2828007ca0a6d5527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } }
I20240502 14:13:35.989539 22732 raft_consensus.cc:388] T 8af0e06b069e424a9ace44e06f12a6d3 P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:35.989696 22732 raft_consensus.cc:482] T 8af0e06b069e424a9ace44e06f12a6d3 P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:35.989655 22731 leader_election.cc:302] T 54ec5887ceee418c9915a1f043978f83 P 61c05b91821b450f9853ce940d18372b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61c05b91821b450f9853ce940d18372b; no voters:
I20240502 14:13:35.989868 22732 raft_consensus.cc:3037] T 8af0e06b069e424a9ace44e06f12a6d3 P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:35.990437 22731 leader_election.cc:288] T 54ec5887ceee418c9915a1f043978f83 P 61c05b91821b450f9853ce940d18372b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:35.990552 22735 raft_consensus.cc:2781] T 54ec5887ceee418c9915a1f043978f83 P 61c05b91821b450f9853ce940d18372b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:35.992210 22731 ts_tablet_manager.cc:1398] T 54ec5887ceee418c9915a1f043978f83 P 61c05b91821b450f9853ce940d18372b: Time spent starting tablet: real 0.017s user 0.011s sys 0.007s
I20240502 14:13:35.992666 22735 raft_consensus.cc:686] T 54ec5887ceee418c9915a1f043978f83 P 61c05b91821b450f9853ce940d18372b [term 1 LEADER]: Becoming Leader. State: Replica: 61c05b91821b450f9853ce940d18372b, State: Running, Role: LEADER
I20240502 14:13:35.993018 22732 raft_consensus.cc:504] T 8af0e06b069e424a9ace44e06f12a6d3 P 0c1d247517e246c2828007ca0a6d5527 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } }
I20240502 14:13:35.993170 22735 consensus_queue.cc:237] T 54ec5887ceee418c9915a1f043978f83 P 61c05b91821b450f9853ce940d18372b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } }
I20240502 14:13:35.993453 22732 leader_election.cc:302] T 8af0e06b069e424a9ace44e06f12a6d3 P 0c1d247517e246c2828007ca0a6d5527 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c1d247517e246c2828007ca0a6d5527; no voters:
I20240502 14:13:35.994071 22732 leader_election.cc:288] T 8af0e06b069e424a9ace44e06f12a6d3 P 0c1d247517e246c2828007ca0a6d5527 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:35.994243 22738 raft_consensus.cc:2781] T 8af0e06b069e424a9ace44e06f12a6d3 P 0c1d247517e246c2828007ca0a6d5527 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:35.995638 22732 ts_tablet_manager.cc:1398] T 8af0e06b069e424a9ace44e06f12a6d3 P 0c1d247517e246c2828007ca0a6d5527: Time spent starting tablet: real 0.017s user 0.017s sys 0.000s
I20240502 14:13:35.996012 22732 tablet_bootstrap.cc:492] T 8a2d3ba227dd43eb941b7e1b3e8445ce P 0c1d247517e246c2828007ca0a6d5527: Bootstrap starting.
I20240502 14:13:35.997579 22732 tablet_bootstrap.cc:654] T 8a2d3ba227dd43eb941b7e1b3e8445ce P 0c1d247517e246c2828007ca0a6d5527: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:35.998842 22738 raft_consensus.cc:686] T 8af0e06b069e424a9ace44e06f12a6d3 P 0c1d247517e246c2828007ca0a6d5527 [term 1 LEADER]: Becoming Leader. State: Replica: 0c1d247517e246c2828007ca0a6d5527, State: Running, Role: LEADER
I20240502 14:13:35.999615 22732 tablet_bootstrap.cc:492] T 8a2d3ba227dd43eb941b7e1b3e8445ce P 0c1d247517e246c2828007ca0a6d5527: No bootstrap required, opened a new log
I20240502 14:13:35.999531 22738 consensus_queue.cc:237] T 8af0e06b069e424a9ace44e06f12a6d3 P 0c1d247517e246c2828007ca0a6d5527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } }
I20240502 14:13:35.999796 22732 ts_tablet_manager.cc:1367] T 8a2d3ba227dd43eb941b7e1b3e8445ce P 0c1d247517e246c2828007ca0a6d5527: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:13:36.000630 22732 raft_consensus.cc:348] T 8a2d3ba227dd43eb941b7e1b3e8445ce P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } }
I20240502 14:13:36.000777 22732 raft_consensus.cc:374] T 8a2d3ba227dd43eb941b7e1b3e8445ce P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:36.000880 22732 raft_consensus.cc:729] T 8a2d3ba227dd43eb941b7e1b3e8445ce P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c1d247517e246c2828007ca0a6d5527, State: Initialized, Role: FOLLOWER
I20240502 14:13:36.001119 22732 consensus_queue.cc:260] T 8a2d3ba227dd43eb941b7e1b3e8445ce P 0c1d247517e246c2828007ca0a6d5527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } }
I20240502 14:13:36.001286 22732 raft_consensus.cc:388] T 8a2d3ba227dd43eb941b7e1b3e8445ce P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:36.001382 22732 raft_consensus.cc:482] T 8a2d3ba227dd43eb941b7e1b3e8445ce P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:36.001499 22732 raft_consensus.cc:3037] T 8a2d3ba227dd43eb941b7e1b3e8445ce P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:36.003474 22732 raft_consensus.cc:504] T 8a2d3ba227dd43eb941b7e1b3e8445ce P 0c1d247517e246c2828007ca0a6d5527 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } }
I20240502 14:13:36.003706 22732 leader_election.cc:302] T 8a2d3ba227dd43eb941b7e1b3e8445ce P 0c1d247517e246c2828007ca0a6d5527 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c1d247517e246c2828007ca0a6d5527; no voters:
I20240502 14:13:36.003890 22732 leader_election.cc:288] T 8a2d3ba227dd43eb941b7e1b3e8445ce P 0c1d247517e246c2828007ca0a6d5527 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:36.004009 22741 raft_consensus.cc:2781] T 8a2d3ba227dd43eb941b7e1b3e8445ce P 0c1d247517e246c2828007ca0a6d5527 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:36.004467 22732 ts_tablet_manager.cc:1398] T 8a2d3ba227dd43eb941b7e1b3e8445ce P 0c1d247517e246c2828007ca0a6d5527: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:13:36.007017 22741 raft_consensus.cc:686] T 8a2d3ba227dd43eb941b7e1b3e8445ce P 0c1d247517e246c2828007ca0a6d5527 [term 1 LEADER]: Becoming Leader. State: Replica: 0c1d247517e246c2828007ca0a6d5527, State: Running, Role: LEADER
I20240502 14:13:36.007431 22741 consensus_queue.cc:237] T 8a2d3ba227dd43eb941b7e1b3e8445ce P 0c1d247517e246c2828007ca0a6d5527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } }
I20240502 14:13:36.008206 21723 catalog_manager.cc:5484] T 54ec5887ceee418c9915a1f043978f83 P 61c05b91821b450f9853ce940d18372b reported cstate change: term changed from 0 to 1, leader changed from <none> to 61c05b91821b450f9853ce940d18372b (127.19.94.194). New cstate: current_term: 1 leader_uuid: "61c05b91821b450f9853ce940d18372b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } }
I20240502 14:13:36.012462 21722 catalog_manager.cc:5484] T 8af0e06b069e424a9ace44e06f12a6d3 P 0c1d247517e246c2828007ca0a6d5527 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193). New cstate: current_term: 1 leader_uuid: "0c1d247517e246c2828007ca0a6d5527" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } }
I20240502 14:13:36.021414 21723 catalog_manager.cc:5484] T 8a2d3ba227dd43eb941b7e1b3e8445ce P 0c1d247517e246c2828007ca0a6d5527 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193). New cstate: current_term: 1 leader_uuid: "0c1d247517e246c2828007ca0a6d5527" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } }
W20240502 14:13:36.366739 22748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:36.367067 22748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:36.374711 22748 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 161.168 ms
time per row: 0.0805838 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 14:13:37.616235 21723 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48804:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:13:37.625279 21975 tablet_service.cc:1467] Processing CreateTablet for tablet 159f343b034a47e88f5d8ecfa0bc282c (DEFAULT_TABLE table=rebalance_test_table_1 [id=0b95f45f4b674b3a81e205cd4f1d6156]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:13:37.625355 21974 tablet_service.cc:1467] Processing CreateTablet for tablet 148d828a22194299a04c2e93003bf379 (DEFAULT_TABLE table=rebalance_test_table_1 [id=0b95f45f4b674b3a81e205cd4f1d6156]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:13:37.625736 21843 tablet_service.cc:1467] Processing CreateTablet for tablet f3119216db7d4c0ea2216744e6f21c5f (DEFAULT_TABLE table=rebalance_test_table_1 [id=0b95f45f4b674b3a81e205cd4f1d6156]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:13:37.625845 21974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 148d828a22194299a04c2e93003bf379. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:37.626052 21975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 159f343b034a47e88f5d8ecfa0bc282c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:37.626102 21843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3119216db7d4c0ea2216744e6f21c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:37.632131 22795 tablet_bootstrap.cc:492] T f3119216db7d4c0ea2216744e6f21c5f P 0c1d247517e246c2828007ca0a6d5527: Bootstrap starting.
I20240502 14:13:37.633065 22796 tablet_bootstrap.cc:492] T 148d828a22194299a04c2e93003bf379 P 61c05b91821b450f9853ce940d18372b: Bootstrap starting.
I20240502 14:13:37.633677 22795 tablet_bootstrap.cc:654] T f3119216db7d4c0ea2216744e6f21c5f P 0c1d247517e246c2828007ca0a6d5527: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:37.634258 22796 tablet_bootstrap.cc:654] T 148d828a22194299a04c2e93003bf379 P 61c05b91821b450f9853ce940d18372b: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:37.635701 22795 tablet_bootstrap.cc:492] T f3119216db7d4c0ea2216744e6f21c5f P 0c1d247517e246c2828007ca0a6d5527: No bootstrap required, opened a new log
I20240502 14:13:37.635838 22795 ts_tablet_manager.cc:1367] T f3119216db7d4c0ea2216744e6f21c5f P 0c1d247517e246c2828007ca0a6d5527: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:13:37.636112 22796 tablet_bootstrap.cc:492] T 148d828a22194299a04c2e93003bf379 P 61c05b91821b450f9853ce940d18372b: No bootstrap required, opened a new log
I20240502 14:13:37.636242 22796 ts_tablet_manager.cc:1367] T 148d828a22194299a04c2e93003bf379 P 61c05b91821b450f9853ce940d18372b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:13:37.636521 22795 raft_consensus.cc:348] T f3119216db7d4c0ea2216744e6f21c5f P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } }
I20240502 14:13:37.636693 22795 raft_consensus.cc:374] T f3119216db7d4c0ea2216744e6f21c5f P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:37.636786 22795 raft_consensus.cc:729] T f3119216db7d4c0ea2216744e6f21c5f P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c1d247517e246c2828007ca0a6d5527, State: Initialized, Role: FOLLOWER
I20240502 14:13:37.636987 22795 consensus_queue.cc:260] T f3119216db7d4c0ea2216744e6f21c5f P 0c1d247517e246c2828007ca0a6d5527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } }
I20240502 14:13:37.637142 22795 raft_consensus.cc:388] T f3119216db7d4c0ea2216744e6f21c5f P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:37.637123 22796 raft_consensus.cc:348] T 148d828a22194299a04c2e93003bf379 P 61c05b91821b450f9853ce940d18372b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } }
I20240502 14:13:37.637233 22795 raft_consensus.cc:482] T f3119216db7d4c0ea2216744e6f21c5f P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:37.637290 22796 raft_consensus.cc:374] T 148d828a22194299a04c2e93003bf379 P 61c05b91821b450f9853ce940d18372b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:37.637343 22795 raft_consensus.cc:3037] T f3119216db7d4c0ea2216744e6f21c5f P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:37.637360 22796 raft_consensus.cc:729] T 148d828a22194299a04c2e93003bf379 P 61c05b91821b450f9853ce940d18372b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61c05b91821b450f9853ce940d18372b, State: Initialized, Role: FOLLOWER
I20240502 14:13:37.637588 22796 consensus_queue.cc:260] T 148d828a22194299a04c2e93003bf379 P 61c05b91821b450f9853ce940d18372b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } }
I20240502 14:13:37.637738 22796 raft_consensus.cc:388] T 148d828a22194299a04c2e93003bf379 P 61c05b91821b450f9853ce940d18372b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:37.637804 22796 raft_consensus.cc:482] T 148d828a22194299a04c2e93003bf379 P 61c05b91821b450f9853ce940d18372b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:37.637881 22796 raft_consensus.cc:3037] T 148d828a22194299a04c2e93003bf379 P 61c05b91821b450f9853ce940d18372b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:37.639469 22795 raft_consensus.cc:504] T f3119216db7d4c0ea2216744e6f21c5f P 0c1d247517e246c2828007ca0a6d5527 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } }
I20240502 14:13:37.639725 22795 leader_election.cc:302] T f3119216db7d4c0ea2216744e6f21c5f P 0c1d247517e246c2828007ca0a6d5527 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c1d247517e246c2828007ca0a6d5527; no voters:
I20240502 14:13:37.640049 22796 raft_consensus.cc:504] T 148d828a22194299a04c2e93003bf379 P 61c05b91821b450f9853ce940d18372b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } }
I20240502 14:13:37.640198 22795 leader_election.cc:288] T f3119216db7d4c0ea2216744e6f21c5f P 0c1d247517e246c2828007ca0a6d5527 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:37.640332 22799 raft_consensus.cc:2781] T f3119216db7d4c0ea2216744e6f21c5f P 0c1d247517e246c2828007ca0a6d5527 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:37.640403 22796 leader_election.cc:302] T 148d828a22194299a04c2e93003bf379 P 61c05b91821b450f9853ce940d18372b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61c05b91821b450f9853ce940d18372b; no voters:
I20240502 14:13:37.641201 22795 ts_tablet_manager.cc:1398] T f3119216db7d4c0ea2216744e6f21c5f P 0c1d247517e246c2828007ca0a6d5527: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240502 14:13:37.641626 22799 raft_consensus.cc:686] T f3119216db7d4c0ea2216744e6f21c5f P 0c1d247517e246c2828007ca0a6d5527 [term 1 LEADER]: Becoming Leader. State: Replica: 0c1d247517e246c2828007ca0a6d5527, State: Running, Role: LEADER
I20240502 14:13:37.641899 22799 consensus_queue.cc:237] T f3119216db7d4c0ea2216744e6f21c5f P 0c1d247517e246c2828007ca0a6d5527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } }
I20240502 14:13:37.643868 22796 leader_election.cc:288] T 148d828a22194299a04c2e93003bf379 P 61c05b91821b450f9853ce940d18372b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:37.644032 22802 raft_consensus.cc:2781] T 148d828a22194299a04c2e93003bf379 P 61c05b91821b450f9853ce940d18372b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:37.644595 21723 catalog_manager.cc:5484] T f3119216db7d4c0ea2216744e6f21c5f P 0c1d247517e246c2828007ca0a6d5527 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193). New cstate: current_term: 1 leader_uuid: "0c1d247517e246c2828007ca0a6d5527" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } }
I20240502 14:13:37.649003 22802 raft_consensus.cc:686] T 148d828a22194299a04c2e93003bf379 P 61c05b91821b450f9853ce940d18372b [term 1 LEADER]: Becoming Leader. State: Replica: 61c05b91821b450f9853ce940d18372b, State: Running, Role: LEADER
I20240502 14:13:37.649065 22796 ts_tablet_manager.cc:1398] T 148d828a22194299a04c2e93003bf379 P 61c05b91821b450f9853ce940d18372b: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20240502 14:13:37.649358 22796 tablet_bootstrap.cc:492] T 159f343b034a47e88f5d8ecfa0bc282c P 61c05b91821b450f9853ce940d18372b: Bootstrap starting.
I20240502 14:13:37.649390 22802 consensus_queue.cc:237] T 148d828a22194299a04c2e93003bf379 P 61c05b91821b450f9853ce940d18372b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } }
I20240502 14:13:37.651090 22796 tablet_bootstrap.cc:654] T 159f343b034a47e88f5d8ecfa0bc282c P 61c05b91821b450f9853ce940d18372b: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:37.652099 21723 catalog_manager.cc:5484] T 148d828a22194299a04c2e93003bf379 P 61c05b91821b450f9853ce940d18372b reported cstate change: term changed from 0 to 1, leader changed from <none> to 61c05b91821b450f9853ce940d18372b (127.19.94.194). New cstate: current_term: 1 leader_uuid: "61c05b91821b450f9853ce940d18372b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } }
I20240502 14:13:37.662333 22796 tablet_bootstrap.cc:492] T 159f343b034a47e88f5d8ecfa0bc282c P 61c05b91821b450f9853ce940d18372b: No bootstrap required, opened a new log
I20240502 14:13:37.662515 22796 ts_tablet_manager.cc:1367] T 159f343b034a47e88f5d8ecfa0bc282c P 61c05b91821b450f9853ce940d18372b: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.006s
I20240502 14:13:37.663161 22796 raft_consensus.cc:348] T 159f343b034a47e88f5d8ecfa0bc282c P 61c05b91821b450f9853ce940d18372b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } }
I20240502 14:13:37.663344 22796 raft_consensus.cc:374] T 159f343b034a47e88f5d8ecfa0bc282c P 61c05b91821b450f9853ce940d18372b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:37.663419 22796 raft_consensus.cc:729] T 159f343b034a47e88f5d8ecfa0bc282c P 61c05b91821b450f9853ce940d18372b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61c05b91821b450f9853ce940d18372b, State: Initialized, Role: FOLLOWER
I20240502 14:13:37.663651 22796 consensus_queue.cc:260] T 159f343b034a47e88f5d8ecfa0bc282c P 61c05b91821b450f9853ce940d18372b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } }
I20240502 14:13:37.663790 22796 raft_consensus.cc:388] T 159f343b034a47e88f5d8ecfa0bc282c P 61c05b91821b450f9853ce940d18372b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:37.663887 22796 raft_consensus.cc:482] T 159f343b034a47e88f5d8ecfa0bc282c P 61c05b91821b450f9853ce940d18372b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:37.663972 22796 raft_consensus.cc:3037] T 159f343b034a47e88f5d8ecfa0bc282c P 61c05b91821b450f9853ce940d18372b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:37.665889 22796 raft_consensus.cc:504] T 159f343b034a47e88f5d8ecfa0bc282c P 61c05b91821b450f9853ce940d18372b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } }
I20240502 14:13:37.666116 22796 leader_election.cc:302] T 159f343b034a47e88f5d8ecfa0bc282c P 61c05b91821b450f9853ce940d18372b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61c05b91821b450f9853ce940d18372b; no voters:
I20240502 14:13:37.666328 22796 leader_election.cc:288] T 159f343b034a47e88f5d8ecfa0bc282c P 61c05b91821b450f9853ce940d18372b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:37.666400 22802 raft_consensus.cc:2781] T 159f343b034a47e88f5d8ecfa0bc282c P 61c05b91821b450f9853ce940d18372b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:37.666661 22802 raft_consensus.cc:686] T 159f343b034a47e88f5d8ecfa0bc282c P 61c05b91821b450f9853ce940d18372b [term 1 LEADER]: Becoming Leader. State: Replica: 61c05b91821b450f9853ce940d18372b, State: Running, Role: LEADER
I20240502 14:13:37.666826 22796 ts_tablet_manager.cc:1398] T 159f343b034a47e88f5d8ecfa0bc282c P 61c05b91821b450f9853ce940d18372b: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:13:37.666923 22802 consensus_queue.cc:237] T 159f343b034a47e88f5d8ecfa0bc282c P 61c05b91821b450f9853ce940d18372b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } }
I20240502 14:13:37.669131 21723 catalog_manager.cc:5484] T 159f343b034a47e88f5d8ecfa0bc282c P 61c05b91821b450f9853ce940d18372b reported cstate change: term changed from 0 to 1, leader changed from <none> to 61c05b91821b450f9853ce940d18372b (127.19.94.194). New cstate: current_term: 1 leader_uuid: "61c05b91821b450f9853ce940d18372b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } }
W20240502 14:13:38.024544 22811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:38.024901 22811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:38.032912 22811 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 126.821 ms
time per row: 0.0634105 ms
I20240502 14:13:38.314945 21720 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48804:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:13:38.324083 21843 tablet_service.cc:1467] Processing CreateTablet for tablet 7253bd03818a4c2f9f0478aebb326273 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1f28b28258954e02b4cf5c905ea2a48c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:13:38.324098 21974 tablet_service.cc:1467] Processing CreateTablet for tablet 906a106c412944e4a049f1491999ad37 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1f28b28258954e02b4cf5c905ea2a48c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:13:38.324410 21844 tablet_service.cc:1467] Processing CreateTablet for tablet 80edabe29f114dd2b3b958fc954e3c2b (DEFAULT_TABLE table=rebalance_test_table_2 [id=1f28b28258954e02b4cf5c905ea2a48c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:13:38.324594 21974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 906a106c412944e4a049f1491999ad37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:38.324602 21843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7253bd03818a4c2f9f0478aebb326273. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:38.324851 21844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80edabe29f114dd2b3b958fc954e3c2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:38.329720 22853 tablet_bootstrap.cc:492] T 906a106c412944e4a049f1491999ad37 P 61c05b91821b450f9853ce940d18372b: Bootstrap starting.
I20240502 14:13:38.331744 22853 tablet_bootstrap.cc:654] T 906a106c412944e4a049f1491999ad37 P 61c05b91821b450f9853ce940d18372b: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:38.332899 22854 tablet_bootstrap.cc:492] T 80edabe29f114dd2b3b958fc954e3c2b P 0c1d247517e246c2828007ca0a6d5527: Bootstrap starting.
I20240502 14:13:38.333716 22853 tablet_bootstrap.cc:492] T 906a106c412944e4a049f1491999ad37 P 61c05b91821b450f9853ce940d18372b: No bootstrap required, opened a new log
I20240502 14:13:38.333878 22853 ts_tablet_manager.cc:1367] T 906a106c412944e4a049f1491999ad37 P 61c05b91821b450f9853ce940d18372b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:13:38.334740 22854 tablet_bootstrap.cc:654] T 80edabe29f114dd2b3b958fc954e3c2b P 0c1d247517e246c2828007ca0a6d5527: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:38.334659 22853 raft_consensus.cc:348] T 906a106c412944e4a049f1491999ad37 P 61c05b91821b450f9853ce940d18372b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } }
I20240502 14:13:38.334846 22853 raft_consensus.cc:374] T 906a106c412944e4a049f1491999ad37 P 61c05b91821b450f9853ce940d18372b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:38.334952 22853 raft_consensus.cc:729] T 906a106c412944e4a049f1491999ad37 P 61c05b91821b450f9853ce940d18372b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61c05b91821b450f9853ce940d18372b, State: Initialized, Role: FOLLOWER
I20240502 14:13:38.335239 22853 consensus_queue.cc:260] T 906a106c412944e4a049f1491999ad37 P 61c05b91821b450f9853ce940d18372b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } }
I20240502 14:13:38.335453 22853 raft_consensus.cc:388] T 906a106c412944e4a049f1491999ad37 P 61c05b91821b450f9853ce940d18372b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:38.335537 22853 raft_consensus.cc:482] T 906a106c412944e4a049f1491999ad37 P 61c05b91821b450f9853ce940d18372b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:38.335656 22853 raft_consensus.cc:3037] T 906a106c412944e4a049f1491999ad37 P 61c05b91821b450f9853ce940d18372b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:38.336818 22854 tablet_bootstrap.cc:492] T 80edabe29f114dd2b3b958fc954e3c2b P 0c1d247517e246c2828007ca0a6d5527: No bootstrap required, opened a new log
I20240502 14:13:38.336983 22854 ts_tablet_manager.cc:1367] T 80edabe29f114dd2b3b958fc954e3c2b P 0c1d247517e246c2828007ca0a6d5527: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:13:38.337731 22854 raft_consensus.cc:348] T 80edabe29f114dd2b3b958fc954e3c2b P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } }
I20240502 14:13:38.337922 22854 raft_consensus.cc:374] T 80edabe29f114dd2b3b958fc954e3c2b P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:38.338022 22854 raft_consensus.cc:729] T 80edabe29f114dd2b3b958fc954e3c2b P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c1d247517e246c2828007ca0a6d5527, State: Initialized, Role: FOLLOWER
I20240502 14:13:38.338260 22853 raft_consensus.cc:504] T 906a106c412944e4a049f1491999ad37 P 61c05b91821b450f9853ce940d18372b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } }
I20240502 14:13:38.338322 22854 consensus_queue.cc:260] T 80edabe29f114dd2b3b958fc954e3c2b P 0c1d247517e246c2828007ca0a6d5527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } }
I20240502 14:13:38.338521 22854 raft_consensus.cc:388] T 80edabe29f114dd2b3b958fc954e3c2b P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:38.338517 22853 leader_election.cc:302] T 906a106c412944e4a049f1491999ad37 P 61c05b91821b450f9853ce940d18372b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61c05b91821b450f9853ce940d18372b; no voters:
I20240502 14:13:38.338601 22854 raft_consensus.cc:482] T 80edabe29f114dd2b3b958fc954e3c2b P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:38.338726 22854 raft_consensus.cc:3037] T 80edabe29f114dd2b3b958fc954e3c2b P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:38.338722 22853 leader_election.cc:288] T 906a106c412944e4a049f1491999ad37 P 61c05b91821b450f9853ce940d18372b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:38.338826 22848 raft_consensus.cc:2781] T 906a106c412944e4a049f1491999ad37 P 61c05b91821b450f9853ce940d18372b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:38.339008 22848 raft_consensus.cc:686] T 906a106c412944e4a049f1491999ad37 P 61c05b91821b450f9853ce940d18372b [term 1 LEADER]: Becoming Leader. State: Replica: 61c05b91821b450f9853ce940d18372b, State: Running, Role: LEADER
I20240502 14:13:38.339208 22853 ts_tablet_manager.cc:1398] T 906a106c412944e4a049f1491999ad37 P 61c05b91821b450f9853ce940d18372b: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240502 14:13:38.339325 22848 consensus_queue.cc:237] T 906a106c412944e4a049f1491999ad37 P 61c05b91821b450f9853ce940d18372b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } }
I20240502 14:13:38.340997 22854 raft_consensus.cc:504] T 80edabe29f114dd2b3b958fc954e3c2b P 0c1d247517e246c2828007ca0a6d5527 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } }
I20240502 14:13:38.341231 22854 leader_election.cc:302] T 80edabe29f114dd2b3b958fc954e3c2b P 0c1d247517e246c2828007ca0a6d5527 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c1d247517e246c2828007ca0a6d5527; no voters:
I20240502 14:13:38.341413 22854 leader_election.cc:288] T 80edabe29f114dd2b3b958fc954e3c2b P 0c1d247517e246c2828007ca0a6d5527 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:38.341495 22843 raft_consensus.cc:2781] T 80edabe29f114dd2b3b958fc954e3c2b P 0c1d247517e246c2828007ca0a6d5527 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:38.341955 21720 catalog_manager.cc:5484] T 906a106c412944e4a049f1491999ad37 P 61c05b91821b450f9853ce940d18372b reported cstate change: term changed from 0 to 1, leader changed from <none> to 61c05b91821b450f9853ce940d18372b (127.19.94.194). New cstate: current_term: 1 leader_uuid: "61c05b91821b450f9853ce940d18372b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } }
I20240502 14:13:38.342703 22854 ts_tablet_manager.cc:1398] T 80edabe29f114dd2b3b958fc954e3c2b P 0c1d247517e246c2828007ca0a6d5527: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20240502 14:13:38.343034 22854 tablet_bootstrap.cc:492] T 7253bd03818a4c2f9f0478aebb326273 P 0c1d247517e246c2828007ca0a6d5527: Bootstrap starting.
I20240502 14:13:38.343765 22843 raft_consensus.cc:686] T 80edabe29f114dd2b3b958fc954e3c2b P 0c1d247517e246c2828007ca0a6d5527 [term 1 LEADER]: Becoming Leader. State: Replica: 0c1d247517e246c2828007ca0a6d5527, State: Running, Role: LEADER
I20240502 14:13:38.344103 22843 consensus_queue.cc:237] T 80edabe29f114dd2b3b958fc954e3c2b P 0c1d247517e246c2828007ca0a6d5527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } }
I20240502 14:13:38.346230 22854 tablet_bootstrap.cc:654] T 7253bd03818a4c2f9f0478aebb326273 P 0c1d247517e246c2828007ca0a6d5527: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:38.346556 21720 catalog_manager.cc:5484] T 80edabe29f114dd2b3b958fc954e3c2b P 0c1d247517e246c2828007ca0a6d5527 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193). New cstate: current_term: 1 leader_uuid: "0c1d247517e246c2828007ca0a6d5527" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } }
I20240502 14:13:38.347911 22854 tablet_bootstrap.cc:492] T 7253bd03818a4c2f9f0478aebb326273 P 0c1d247517e246c2828007ca0a6d5527: No bootstrap required, opened a new log
I20240502 14:13:38.348037 22854 ts_tablet_manager.cc:1367] T 7253bd03818a4c2f9f0478aebb326273 P 0c1d247517e246c2828007ca0a6d5527: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240502 14:13:38.348651 22854 raft_consensus.cc:348] T 7253bd03818a4c2f9f0478aebb326273 P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } }
I20240502 14:13:38.348845 22854 raft_consensus.cc:374] T 7253bd03818a4c2f9f0478aebb326273 P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:38.348933 22854 raft_consensus.cc:729] T 7253bd03818a4c2f9f0478aebb326273 P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c1d247517e246c2828007ca0a6d5527, State: Initialized, Role: FOLLOWER
I20240502 14:13:38.349167 22854 consensus_queue.cc:260] T 7253bd03818a4c2f9f0478aebb326273 P 0c1d247517e246c2828007ca0a6d5527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } }
I20240502 14:13:38.349306 22854 raft_consensus.cc:388] T 7253bd03818a4c2f9f0478aebb326273 P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:38.349413 22854 raft_consensus.cc:482] T 7253bd03818a4c2f9f0478aebb326273 P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:38.350020 22854 raft_consensus.cc:3037] T 7253bd03818a4c2f9f0478aebb326273 P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:38.351949 22854 raft_consensus.cc:504] T 7253bd03818a4c2f9f0478aebb326273 P 0c1d247517e246c2828007ca0a6d5527 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } }
I20240502 14:13:38.352142 22854 leader_election.cc:302] T 7253bd03818a4c2f9f0478aebb326273 P 0c1d247517e246c2828007ca0a6d5527 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c1d247517e246c2828007ca0a6d5527; no voters:
I20240502 14:13:38.352309 22854 leader_election.cc:288] T 7253bd03818a4c2f9f0478aebb326273 P 0c1d247517e246c2828007ca0a6d5527 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:38.352372 22843 raft_consensus.cc:2781] T 7253bd03818a4c2f9f0478aebb326273 P 0c1d247517e246c2828007ca0a6d5527 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:38.352550 22843 raft_consensus.cc:686] T 7253bd03818a4c2f9f0478aebb326273 P 0c1d247517e246c2828007ca0a6d5527 [term 1 LEADER]: Becoming Leader. State: Replica: 0c1d247517e246c2828007ca0a6d5527, State: Running, Role: LEADER
I20240502 14:13:38.352847 22854 ts_tablet_manager.cc:1398] T 7253bd03818a4c2f9f0478aebb326273 P 0c1d247517e246c2828007ca0a6d5527: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 14:13:38.352788 22843 consensus_queue.cc:237] T 7253bd03818a4c2f9f0478aebb326273 P 0c1d247517e246c2828007ca0a6d5527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } }
I20240502 14:13:38.354823 21720 catalog_manager.cc:5484] T 7253bd03818a4c2f9f0478aebb326273 P 0c1d247517e246c2828007ca0a6d5527 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193). New cstate: current_term: 1 leader_uuid: "0c1d247517e246c2828007ca0a6d5527" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } }
W20240502 14:13:38.716156 22863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:38.716571 22863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:38.724807 22863 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 112.167 ms
time per row: 0.0560834 ms
I20240502 14:13:38.979158 21722 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48804:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:13:38.988683 21974 tablet_service.cc:1467] Processing CreateTablet for tablet 48dfb482c48d408aa359c0b948237fd6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a069a57795f6421096699da7bddb7a1c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:13:38.988936 21975 tablet_service.cc:1467] Processing CreateTablet for tablet 2ffa22a372c143bc87194122a407425c (DEFAULT_TABLE table=rebalance_test_table_3 [id=a069a57795f6421096699da7bddb7a1c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:13:38.989235 21974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48dfb482c48d408aa359c0b948237fd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:38.989224 21844 tablet_service.cc:1467] Processing CreateTablet for tablet ab54535b58be4e518340f33449bcdb26 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a069a57795f6421096699da7bddb7a1c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:13:38.989523 21975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ffa22a372c143bc87194122a407425c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:38.989650 21844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab54535b58be4e518340f33449bcdb26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:38.994621 22906 tablet_bootstrap.cc:492] T ab54535b58be4e518340f33449bcdb26 P 0c1d247517e246c2828007ca0a6d5527: Bootstrap starting.
I20240502 14:13:38.996446 22906 tablet_bootstrap.cc:654] T ab54535b58be4e518340f33449bcdb26 P 0c1d247517e246c2828007ca0a6d5527: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:38.996894 22907 tablet_bootstrap.cc:492] T 48dfb482c48d408aa359c0b948237fd6 P 61c05b91821b450f9853ce940d18372b: Bootstrap starting.
I20240502 14:13:38.998543 22907 tablet_bootstrap.cc:654] T 48dfb482c48d408aa359c0b948237fd6 P 61c05b91821b450f9853ce940d18372b: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:38.998549 22906 tablet_bootstrap.cc:492] T ab54535b58be4e518340f33449bcdb26 P 0c1d247517e246c2828007ca0a6d5527: No bootstrap required, opened a new log
I20240502 14:13:38.998718 22906 ts_tablet_manager.cc:1367] T ab54535b58be4e518340f33449bcdb26 P 0c1d247517e246c2828007ca0a6d5527: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:13:38.999485 22906 raft_consensus.cc:348] T ab54535b58be4e518340f33449bcdb26 P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } }
I20240502 14:13:38.999656 22906 raft_consensus.cc:374] T ab54535b58be4e518340f33449bcdb26 P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:38.999748 22906 raft_consensus.cc:729] T ab54535b58be4e518340f33449bcdb26 P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c1d247517e246c2828007ca0a6d5527, State: Initialized, Role: FOLLOWER
I20240502 14:13:38.999960 22906 consensus_queue.cc:260] T ab54535b58be4e518340f33449bcdb26 P 0c1d247517e246c2828007ca0a6d5527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } }
I20240502 14:13:39.000118 22906 raft_consensus.cc:388] T ab54535b58be4e518340f33449bcdb26 P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:39.000190 22906 raft_consensus.cc:482] T ab54535b58be4e518340f33449bcdb26 P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:39.000293 22906 raft_consensus.cc:3037] T ab54535b58be4e518340f33449bcdb26 P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:39.000420 22907 tablet_bootstrap.cc:492] T 48dfb482c48d408aa359c0b948237fd6 P 61c05b91821b450f9853ce940d18372b: No bootstrap required, opened a new log
I20240502 14:13:39.000553 22907 ts_tablet_manager.cc:1367] T 48dfb482c48d408aa359c0b948237fd6 P 61c05b91821b450f9853ce940d18372b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20240502 14:13:39.001117 22907 raft_consensus.cc:348] T 48dfb482c48d408aa359c0b948237fd6 P 61c05b91821b450f9853ce940d18372b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } }
I20240502 14:13:39.001256 22907 raft_consensus.cc:374] T 48dfb482c48d408aa359c0b948237fd6 P 61c05b91821b450f9853ce940d18372b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:39.001350 22907 raft_consensus.cc:729] T 48dfb482c48d408aa359c0b948237fd6 P 61c05b91821b450f9853ce940d18372b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61c05b91821b450f9853ce940d18372b, State: Initialized, Role: FOLLOWER
I20240502 14:13:39.001569 22907 consensus_queue.cc:260] T 48dfb482c48d408aa359c0b948237fd6 P 61c05b91821b450f9853ce940d18372b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } }
I20240502 14:13:39.001740 22907 raft_consensus.cc:388] T 48dfb482c48d408aa359c0b948237fd6 P 61c05b91821b450f9853ce940d18372b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:39.001838 22907 raft_consensus.cc:482] T 48dfb482c48d408aa359c0b948237fd6 P 61c05b91821b450f9853ce940d18372b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:39.001919 22907 raft_consensus.cc:3037] T 48dfb482c48d408aa359c0b948237fd6 P 61c05b91821b450f9853ce940d18372b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:39.002506 22906 raft_consensus.cc:504] T ab54535b58be4e518340f33449bcdb26 P 0c1d247517e246c2828007ca0a6d5527 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } }
I20240502 14:13:39.002785 22906 leader_election.cc:302] T ab54535b58be4e518340f33449bcdb26 P 0c1d247517e246c2828007ca0a6d5527 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c1d247517e246c2828007ca0a6d5527; no voters:
I20240502 14:13:39.002980 22906 leader_election.cc:288] T ab54535b58be4e518340f33449bcdb26 P 0c1d247517e246c2828007ca0a6d5527 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:39.003110 22843 raft_consensus.cc:2781] T ab54535b58be4e518340f33449bcdb26 P 0c1d247517e246c2828007ca0a6d5527 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:39.003484 22906 ts_tablet_manager.cc:1398] T ab54535b58be4e518340f33449bcdb26 P 0c1d247517e246c2828007ca0a6d5527: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240502 14:13:39.004249 22907 raft_consensus.cc:504] T 48dfb482c48d408aa359c0b948237fd6 P 61c05b91821b450f9853ce940d18372b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } }
I20240502 14:13:39.004503 22907 leader_election.cc:302] T 48dfb482c48d408aa359c0b948237fd6 P 61c05b91821b450f9853ce940d18372b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61c05b91821b450f9853ce940d18372b; no voters:
I20240502 14:13:39.004645 22843 raft_consensus.cc:686] T ab54535b58be4e518340f33449bcdb26 P 0c1d247517e246c2828007ca0a6d5527 [term 1 LEADER]: Becoming Leader. State: Replica: 0c1d247517e246c2828007ca0a6d5527, State: Running, Role: LEADER
I20240502 14:13:39.004709 22907 leader_election.cc:288] T 48dfb482c48d408aa359c0b948237fd6 P 61c05b91821b450f9853ce940d18372b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:39.004814 22895 raft_consensus.cc:2781] T 48dfb482c48d408aa359c0b948237fd6 P 61c05b91821b450f9853ce940d18372b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:39.004894 22843 consensus_queue.cc:237] T ab54535b58be4e518340f33449bcdb26 P 0c1d247517e246c2828007ca0a6d5527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } }
I20240502 14:13:39.005903 22907 ts_tablet_manager.cc:1398] T 48dfb482c48d408aa359c0b948237fd6 P 61c05b91821b450f9853ce940d18372b: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20240502 14:13:39.006204 22907 tablet_bootstrap.cc:492] T 2ffa22a372c143bc87194122a407425c P 61c05b91821b450f9853ce940d18372b: Bootstrap starting.
I20240502 14:13:39.008116 22907 tablet_bootstrap.cc:654] T 2ffa22a372c143bc87194122a407425c P 61c05b91821b450f9853ce940d18372b: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:39.008939 21722 catalog_manager.cc:5484] T ab54535b58be4e518340f33449bcdb26 P 0c1d247517e246c2828007ca0a6d5527 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193). New cstate: current_term: 1 leader_uuid: "0c1d247517e246c2828007ca0a6d5527" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } }
I20240502 14:13:39.009681 22895 raft_consensus.cc:686] T 48dfb482c48d408aa359c0b948237fd6 P 61c05b91821b450f9853ce940d18372b [term 1 LEADER]: Becoming Leader. State: Replica: 61c05b91821b450f9853ce940d18372b, State: Running, Role: LEADER
I20240502 14:13:39.010010 22895 consensus_queue.cc:237] T 48dfb482c48d408aa359c0b948237fd6 P 61c05b91821b450f9853ce940d18372b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } }
I20240502 14:13:39.010599 22907 tablet_bootstrap.cc:492] T 2ffa22a372c143bc87194122a407425c P 61c05b91821b450f9853ce940d18372b: No bootstrap required, opened a new log
I20240502 14:13:39.011273 22907 ts_tablet_manager.cc:1367] T 2ffa22a372c143bc87194122a407425c P 61c05b91821b450f9853ce940d18372b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 14:13:39.012632 21723 catalog_manager.cc:5484] T 48dfb482c48d408aa359c0b948237fd6 P 61c05b91821b450f9853ce940d18372b reported cstate change: term changed from 0 to 1, leader changed from <none> to 61c05b91821b450f9853ce940d18372b (127.19.94.194). New cstate: current_term: 1 leader_uuid: "61c05b91821b450f9853ce940d18372b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } }
I20240502 14:13:39.013362 22907 raft_consensus.cc:348] T 2ffa22a372c143bc87194122a407425c P 61c05b91821b450f9853ce940d18372b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } }
I20240502 14:13:39.013552 22907 raft_consensus.cc:374] T 2ffa22a372c143bc87194122a407425c P 61c05b91821b450f9853ce940d18372b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:39.013610 22907 raft_consensus.cc:729] T 2ffa22a372c143bc87194122a407425c P 61c05b91821b450f9853ce940d18372b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61c05b91821b450f9853ce940d18372b, State: Initialized, Role: FOLLOWER
I20240502 14:13:39.013828 22907 consensus_queue.cc:260] T 2ffa22a372c143bc87194122a407425c P 61c05b91821b450f9853ce940d18372b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } }
I20240502 14:13:39.013952 22907 raft_consensus.cc:388] T 2ffa22a372c143bc87194122a407425c P 61c05b91821b450f9853ce940d18372b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:39.014078 22907 raft_consensus.cc:482] T 2ffa22a372c143bc87194122a407425c P 61c05b91821b450f9853ce940d18372b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:39.014189 22907 raft_consensus.cc:3037] T 2ffa22a372c143bc87194122a407425c P 61c05b91821b450f9853ce940d18372b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:39.016620 22907 raft_consensus.cc:504] T 2ffa22a372c143bc87194122a407425c P 61c05b91821b450f9853ce940d18372b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } }
I20240502 14:13:39.016842 22907 leader_election.cc:302] T 2ffa22a372c143bc87194122a407425c P 61c05b91821b450f9853ce940d18372b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61c05b91821b450f9853ce940d18372b; no voters:
I20240502 14:13:39.017027 22907 leader_election.cc:288] T 2ffa22a372c143bc87194122a407425c P 61c05b91821b450f9853ce940d18372b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:39.017071 22895 raft_consensus.cc:2781] T 2ffa22a372c143bc87194122a407425c P 61c05b91821b450f9853ce940d18372b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:39.017230 22895 raft_consensus.cc:686] T 2ffa22a372c143bc87194122a407425c P 61c05b91821b450f9853ce940d18372b [term 1 LEADER]: Becoming Leader. State: Replica: 61c05b91821b450f9853ce940d18372b, State: Running, Role: LEADER
I20240502 14:13:39.017431 22895 consensus_queue.cc:237] T 2ffa22a372c143bc87194122a407425c P 61c05b91821b450f9853ce940d18372b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } }
I20240502 14:13:39.019008 22907 ts_tablet_manager.cc:1398] T 2ffa22a372c143bc87194122a407425c P 61c05b91821b450f9853ce940d18372b: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240502 14:13:39.019505 21722 catalog_manager.cc:5484] T 2ffa22a372c143bc87194122a407425c P 61c05b91821b450f9853ce940d18372b reported cstate change: term changed from 0 to 1, leader changed from <none> to 61c05b91821b450f9853ce940d18372b (127.19.94.194). New cstate: current_term: 1 leader_uuid: "61c05b91821b450f9853ce940d18372b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } }
W20240502 14:13:39.379169 22917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:39.379565 22917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:39.387523 22917 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 163.633 ms
time per row: 0.0818166 ms
I20240502 14:13:39.705469 21723 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48804:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:13:39.714890 21844 tablet_service.cc:1467] Processing CreateTablet for tablet 4cc5c4909e7343578a87c1b91ed3d756 (DEFAULT_TABLE table=rebalance_test_table_4 [id=fe06048058fb45dcb952409c2f9a3754]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:13:39.715060 21974 tablet_service.cc:1467] Processing CreateTablet for tablet 9597a33811ef4247a4c0ecb76d0afbaf (DEFAULT_TABLE table=rebalance_test_table_4 [id=fe06048058fb45dcb952409c2f9a3754]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:13:39.715255 21843 tablet_service.cc:1467] Processing CreateTablet for tablet 424a2170ba374ed1982e550e8f7ebeea (DEFAULT_TABLE table=rebalance_test_table_4 [id=fe06048058fb45dcb952409c2f9a3754]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:13:39.715456 21844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cc5c4909e7343578a87c1b91ed3d756. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:39.715545 21974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9597a33811ef4247a4c0ecb76d0afbaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:39.715853 21843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 424a2170ba374ed1982e550e8f7ebeea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:39.722587 22963 tablet_bootstrap.cc:492] T 9597a33811ef4247a4c0ecb76d0afbaf P 61c05b91821b450f9853ce940d18372b: Bootstrap starting.
I20240502 14:13:39.723371 22964 tablet_bootstrap.cc:492] T 424a2170ba374ed1982e550e8f7ebeea P 0c1d247517e246c2828007ca0a6d5527: Bootstrap starting.
I20240502 14:13:39.724731 22963 tablet_bootstrap.cc:654] T 9597a33811ef4247a4c0ecb76d0afbaf P 61c05b91821b450f9853ce940d18372b: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:39.725327 22964 tablet_bootstrap.cc:654] T 424a2170ba374ed1982e550e8f7ebeea P 0c1d247517e246c2828007ca0a6d5527: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:39.727334 22963 tablet_bootstrap.cc:492] T 9597a33811ef4247a4c0ecb76d0afbaf P 61c05b91821b450f9853ce940d18372b: No bootstrap required, opened a new log
I20240502 14:13:39.727547 22963 ts_tablet_manager.cc:1367] T 9597a33811ef4247a4c0ecb76d0afbaf P 61c05b91821b450f9853ce940d18372b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20240502 14:13:39.728365 22963 raft_consensus.cc:348] T 9597a33811ef4247a4c0ecb76d0afbaf P 61c05b91821b450f9853ce940d18372b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } }
I20240502 14:13:39.728611 22963 raft_consensus.cc:374] T 9597a33811ef4247a4c0ecb76d0afbaf P 61c05b91821b450f9853ce940d18372b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:39.728713 22964 tablet_bootstrap.cc:492] T 424a2170ba374ed1982e550e8f7ebeea P 0c1d247517e246c2828007ca0a6d5527: No bootstrap required, opened a new log
I20240502 14:13:39.728718 22963 raft_consensus.cc:729] T 9597a33811ef4247a4c0ecb76d0afbaf P 61c05b91821b450f9853ce940d18372b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61c05b91821b450f9853ce940d18372b, State: Initialized, Role: FOLLOWER
I20240502 14:13:39.728886 22964 ts_tablet_manager.cc:1367] T 424a2170ba374ed1982e550e8f7ebeea P 0c1d247517e246c2828007ca0a6d5527: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240502 14:13:39.728991 22963 consensus_queue.cc:260] T 9597a33811ef4247a4c0ecb76d0afbaf P 61c05b91821b450f9853ce940d18372b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } }
I20240502 14:13:39.729216 22963 raft_consensus.cc:388] T 9597a33811ef4247a4c0ecb76d0afbaf P 61c05b91821b450f9853ce940d18372b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:39.729310 22963 raft_consensus.cc:482] T 9597a33811ef4247a4c0ecb76d0afbaf P 61c05b91821b450f9853ce940d18372b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:39.729429 22963 raft_consensus.cc:3037] T 9597a33811ef4247a4c0ecb76d0afbaf P 61c05b91821b450f9853ce940d18372b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:39.729772 22964 raft_consensus.cc:348] T 424a2170ba374ed1982e550e8f7ebeea P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } }
I20240502 14:13:39.729961 22964 raft_consensus.cc:374] T 424a2170ba374ed1982e550e8f7ebeea P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:39.730054 22964 raft_consensus.cc:729] T 424a2170ba374ed1982e550e8f7ebeea P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c1d247517e246c2828007ca0a6d5527, State: Initialized, Role: FOLLOWER
I20240502 14:13:39.730322 22964 consensus_queue.cc:260] T 424a2170ba374ed1982e550e8f7ebeea P 0c1d247517e246c2828007ca0a6d5527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } }
I20240502 14:13:39.730526 22964 raft_consensus.cc:388] T 424a2170ba374ed1982e550e8f7ebeea P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:39.730613 22964 raft_consensus.cc:482] T 424a2170ba374ed1982e550e8f7ebeea P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:39.730736 22964 raft_consensus.cc:3037] T 424a2170ba374ed1982e550e8f7ebeea P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:39.732374 22963 raft_consensus.cc:504] T 9597a33811ef4247a4c0ecb76d0afbaf P 61c05b91821b450f9853ce940d18372b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } }
I20240502 14:13:39.732689 22963 leader_election.cc:302] T 9597a33811ef4247a4c0ecb76d0afbaf P 61c05b91821b450f9853ce940d18372b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61c05b91821b450f9853ce940d18372b; no voters:
I20240502 14:13:39.732928 22963 leader_election.cc:288] T 9597a33811ef4247a4c0ecb76d0afbaf P 61c05b91821b450f9853ce940d18372b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:39.733096 22952 raft_consensus.cc:2781] T 9597a33811ef4247a4c0ecb76d0afbaf P 61c05b91821b450f9853ce940d18372b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:39.733357 22952 raft_consensus.cc:686] T 9597a33811ef4247a4c0ecb76d0afbaf P 61c05b91821b450f9853ce940d18372b [term 1 LEADER]: Becoming Leader. State: Replica: 61c05b91821b450f9853ce940d18372b, State: Running, Role: LEADER
I20240502 14:13:39.733664 22952 consensus_queue.cc:237] T 9597a33811ef4247a4c0ecb76d0afbaf P 61c05b91821b450f9853ce940d18372b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } }
I20240502 14:13:39.733668 22964 raft_consensus.cc:504] T 424a2170ba374ed1982e550e8f7ebeea P 0c1d247517e246c2828007ca0a6d5527 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } }
I20240502 14:13:39.733870 22963 ts_tablet_manager.cc:1398] T 9597a33811ef4247a4c0ecb76d0afbaf P 61c05b91821b450f9853ce940d18372b: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20240502 14:13:39.733942 22964 leader_election.cc:302] T 424a2170ba374ed1982e550e8f7ebeea P 0c1d247517e246c2828007ca0a6d5527 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c1d247517e246c2828007ca0a6d5527; no voters:
I20240502 14:13:39.734238 22964 leader_election.cc:288] T 424a2170ba374ed1982e550e8f7ebeea P 0c1d247517e246c2828007ca0a6d5527 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:39.734328 22950 raft_consensus.cc:2781] T 424a2170ba374ed1982e550e8f7ebeea P 0c1d247517e246c2828007ca0a6d5527 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:39.735591 22964 ts_tablet_manager.cc:1398] T 424a2170ba374ed1982e550e8f7ebeea P 0c1d247517e246c2828007ca0a6d5527: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20240502 14:13:39.735992 22964 tablet_bootstrap.cc:492] T 4cc5c4909e7343578a87c1b91ed3d756 P 0c1d247517e246c2828007ca0a6d5527: Bootstrap starting.
I20240502 14:13:39.736819 22950 raft_consensus.cc:686] T 424a2170ba374ed1982e550e8f7ebeea P 0c1d247517e246c2828007ca0a6d5527 [term 1 LEADER]: Becoming Leader. State: Replica: 0c1d247517e246c2828007ca0a6d5527, State: Running, Role: LEADER
I20240502 14:13:39.737152 22950 consensus_queue.cc:237] T 424a2170ba374ed1982e550e8f7ebeea P 0c1d247517e246c2828007ca0a6d5527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } }
I20240502 14:13:39.737902 22964 tablet_bootstrap.cc:654] T 4cc5c4909e7343578a87c1b91ed3d756 P 0c1d247517e246c2828007ca0a6d5527: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:39.738472 21723 catalog_manager.cc:5484] T 9597a33811ef4247a4c0ecb76d0afbaf P 61c05b91821b450f9853ce940d18372b reported cstate change: term changed from 0 to 1, leader changed from <none> to 61c05b91821b450f9853ce940d18372b (127.19.94.194). New cstate: current_term: 1 leader_uuid: "61c05b91821b450f9853ce940d18372b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } }
I20240502 14:13:39.740689 21722 catalog_manager.cc:5484] T 424a2170ba374ed1982e550e8f7ebeea P 0c1d247517e246c2828007ca0a6d5527 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193). New cstate: current_term: 1 leader_uuid: "0c1d247517e246c2828007ca0a6d5527" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } }
I20240502 14:13:39.743238 22964 tablet_bootstrap.cc:492] T 4cc5c4909e7343578a87c1b91ed3d756 P 0c1d247517e246c2828007ca0a6d5527: No bootstrap required, opened a new log
I20240502 14:13:39.743435 22964 ts_tablet_manager.cc:1367] T 4cc5c4909e7343578a87c1b91ed3d756 P 0c1d247517e246c2828007ca0a6d5527: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240502 14:13:39.744278 22964 raft_consensus.cc:348] T 4cc5c4909e7343578a87c1b91ed3d756 P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } }
I20240502 14:13:39.744529 22964 raft_consensus.cc:374] T 4cc5c4909e7343578a87c1b91ed3d756 P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:39.744616 22964 raft_consensus.cc:729] T 4cc5c4909e7343578a87c1b91ed3d756 P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c1d247517e246c2828007ca0a6d5527, State: Initialized, Role: FOLLOWER
I20240502 14:13:39.744882 22964 consensus_queue.cc:260] T 4cc5c4909e7343578a87c1b91ed3d756 P 0c1d247517e246c2828007ca0a6d5527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } }
I20240502 14:13:39.745079 22964 raft_consensus.cc:388] T 4cc5c4909e7343578a87c1b91ed3d756 P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:39.745200 22964 raft_consensus.cc:482] T 4cc5c4909e7343578a87c1b91ed3d756 P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:39.745302 22964 raft_consensus.cc:3037] T 4cc5c4909e7343578a87c1b91ed3d756 P 0c1d247517e246c2828007ca0a6d5527 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:39.747383 22964 raft_consensus.cc:504] T 4cc5c4909e7343578a87c1b91ed3d756 P 0c1d247517e246c2828007ca0a6d5527 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } }
I20240502 14:13:39.747649 22964 leader_election.cc:302] T 4cc5c4909e7343578a87c1b91ed3d756 P 0c1d247517e246c2828007ca0a6d5527 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c1d247517e246c2828007ca0a6d5527; no voters:
I20240502 14:13:39.747861 22964 leader_election.cc:288] T 4cc5c4909e7343578a87c1b91ed3d756 P 0c1d247517e246c2828007ca0a6d5527 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:39.747953 22950 raft_consensus.cc:2781] T 4cc5c4909e7343578a87c1b91ed3d756 P 0c1d247517e246c2828007ca0a6d5527 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:39.748114 22950 raft_consensus.cc:686] T 4cc5c4909e7343578a87c1b91ed3d756 P 0c1d247517e246c2828007ca0a6d5527 [term 1 LEADER]: Becoming Leader. State: Replica: 0c1d247517e246c2828007ca0a6d5527, State: Running, Role: LEADER
I20240502 14:13:39.748417 22964 ts_tablet_manager.cc:1398] T 4cc5c4909e7343578a87c1b91ed3d756 P 0c1d247517e246c2828007ca0a6d5527: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:13:39.748354 22950 consensus_queue.cc:237] T 4cc5c4909e7343578a87c1b91ed3d756 P 0c1d247517e246c2828007ca0a6d5527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } }
I20240502 14:13:39.752626 21715 catalog_manager.cc:5484] T 4cc5c4909e7343578a87c1b91ed3d756 P 0c1d247517e246c2828007ca0a6d5527 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193). New cstate: current_term: 1 leader_uuid: "0c1d247517e246c2828007ca0a6d5527" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } }
W20240502 14:13:40.123145 22973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:40.123557 22973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:40.131669 22973 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 145.924 ms
time per row: 0.0729619 ms
I20240502 14:13:40.428479 19835 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_zeJNl/build/asan/bin/kudu
/tmp/dist-test-task_zeJNl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.94.195:39713
--local_ip_for_outbound_sockets=127.19.94.195
--tserver_master_addrs=127.19.94.254:40967
--webserver_port=45307
--webserver_interface=127.19.94.195
--builtin_ntp_servers=127.19.94.212:39595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 14:13:40.787060 23016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:40.787431 23016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:40.787596 23016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:40.796839 23016 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:13:40.797030 23016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:40.797159 23016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.94.195
I20240502 14:13:40.808756 23016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.94.212:39595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.94.195:39713
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.19.94.195
--webserver_port=45307
--tserver_master_addrs=127.19.94.254:40967
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.94.195
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:40.810178 23016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:40.812134 23016 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:40.828953 23021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:40.829696 23022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:40.830458 23024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:40.831319 23016 server_base.cc:1034] running on GCE node
I20240502 14:13:40.832073 23016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:40.833361 23016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:40.834594 23016 hybrid_clock.cc:648] HybridClock initialized: now 1714659220834552 us; error 37 us; skew 500 ppm
I20240502 14:13:40.834957 23016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:40.838064 23016 webserver.cc:460] Webserver started at http://127.19.94.195:45307/ using document root <none> and password file <none>
I20240502 14:13:40.838995 23016 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:40.839154 23016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:40.845662 23016 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20240502 14:13:40.849704 23030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:40.851624 23016 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20240502 14:13:40.851852 23016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ff15cd84fa6640b8921bbf05dc65612e"
format_stamp: "Formatted at 2024-05-02 14:13:29 on dist-test-slave-3j2v"
I20240502 14:13:40.852869 23016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:40.871908 23016 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:40.873409 23016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:40.873816 23016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:40.875351 23016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:40.877233 23016 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:40.877331 23016 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:40.877436 23016 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:40.877482 23016 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:40.913185 23016 rpc_server.cc:307] RPC server started. Bound to: 127.19.94.195:39713
I20240502 14:13:40.913250 23142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.94.195:39713 every 8 connection(s)
I20240502 14:13:40.914803 23016 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 14:13:40.920140 19835 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_zeJNl/build/asan/bin/kudu as pid 23016
I20240502 14:13:40.920975 19835 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_zeJNl/build/asan/bin/kudu
/tmp/dist-test-task_zeJNl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.94.196:41835
--local_ip_for_outbound_sockets=127.19.94.196
--tserver_master_addrs=127.19.94.254:40967
--webserver_port=37395
--webserver_interface=127.19.94.196
--builtin_ntp_servers=127.19.94.212:39595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:13:40.932111 23143 heartbeater.cc:347] Connected to a master server at 127.19.94.254:40967
I20240502 14:13:40.932544 23143 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:40.933409 23143 heartbeater.cc:511] Master 127.19.94.254:40967 requested a full tablet report, sending...
I20240502 14:13:40.935411 21722 ts_manager.cc:194] Re-registered known tserver with Master: ff15cd84fa6640b8921bbf05dc65612e (127.19.94.195:39713)
I20240502 14:13:40.936822 21722 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.94.195:50605
W20240502 14:13:41.276437 23147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:41.276787 23147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:41.276895 23147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:41.285030 23147 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:13:41.285174 23147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:41.285301 23147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.94.196
I20240502 14:13:41.295069 23147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.94.212:39595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.94.196:41835
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.19.94.196
--webserver_port=37395
--tserver_master_addrs=127.19.94.254:40967
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.94.196
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:41.296505 23147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:41.298245 23147 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:41.316243 23153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:41.316810 23152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:41.317464 23155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:41.318603 23147 server_base.cc:1034] running on GCE node
I20240502 14:13:41.319321 23147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:41.320619 23147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:41.322088 23147 hybrid_clock.cc:648] HybridClock initialized: now 1714659221321936 us; error 42 us; skew 500 ppm
I20240502 14:13:41.322497 23147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:41.325310 23147 webserver.cc:460] Webserver started at http://127.19.94.196:37395/ using document root <none> and password file <none>
I20240502 14:13:41.326157 23147 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:41.326308 23147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:41.332161 23147 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 14:13:41.335623 23161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:41.337381 23147 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240502 14:13:41.337602 23147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "f8fbf60396bb443697ce265de0eb063e"
format_stamp: "Formatted at 2024-05-02 14:13:29 on dist-test-slave-3j2v"
I20240502 14:13:41.338522 23147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:41.363739 23147 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:41.364847 23147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:41.365178 23147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:41.366240 23147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:41.367923 23147 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:41.368006 23147 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:41.368160 23147 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:41.368216 23147 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:41.402443 23147 rpc_server.cc:307] RPC server started. Bound to: 127.19.94.196:41835
I20240502 14:13:41.402475 23274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.94.196:41835 every 8 connection(s)
I20240502 14:13:41.404028 23147 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 14:13:41.405426 19835 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_zeJNl/build/asan/bin/kudu as pid 23147
I20240502 14:13:41.406164 19835 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_zeJNl/build/asan/bin/kudu
/tmp/dist-test-task_zeJNl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.94.197:33817
--local_ip_for_outbound_sockets=127.19.94.197
--tserver_master_addrs=127.19.94.254:40967
--webserver_port=37515
--webserver_interface=127.19.94.197
--builtin_ntp_servers=127.19.94.212:39595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:13:41.425228 23275 heartbeater.cc:347] Connected to a master server at 127.19.94.254:40967
I20240502 14:13:41.425576 23275 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:41.426394 23275 heartbeater.cc:511] Master 127.19.94.254:40967 requested a full tablet report, sending...
I20240502 14:13:41.427996 21722 ts_manager.cc:194] Re-registered known tserver with Master: f8fbf60396bb443697ce265de0eb063e (127.19.94.196:41835)
I20240502 14:13:41.429180 21722 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.94.196:49889
W20240502 14:13:41.758340 23278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:41.758792 23278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:41.758953 23278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:41.767158 23278 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:13:41.767299 23278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:41.767417 23278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.94.197
I20240502 14:13:41.776837 23278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.94.212:39595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.94.197:33817
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.19.94.197
--webserver_port=37515
--tserver_master_addrs=127.19.94.254:40967
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.94.197
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:41.778211 23278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:41.780009 23278 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:41.793530 23284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:41.794389 23283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:41.795118 23286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:41.795785 23278 server_base.cc:1034] running on GCE node
I20240502 14:13:41.796618 23278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:41.797621 23278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:41.798882 23278 hybrid_clock.cc:648] HybridClock initialized: now 1714659221798804 us; error 51 us; skew 500 ppm
I20240502 14:13:41.799240 23278 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:41.802125 23278 webserver.cc:460] Webserver started at http://127.19.94.197:37515/ using document root <none> and password file <none>
I20240502 14:13:41.802982 23278 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:41.803112 23278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:41.810146 23278 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20240502 14:13:41.814615 23292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:41.816704 23278 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.001s
I20240502 14:13:41.816937 23278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "92646999d5c64d1aae6f165593b7d14d"
format_stamp: "Formatted at 2024-05-02 14:13:30 on dist-test-slave-3j2v"
I20240502 14:13:41.817932 23278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:41.833204 23278 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:41.834149 23278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:41.834439 23278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:41.835450 23278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:41.837870 23299 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240502 14:13:41.844211 23278 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240502 14:13:41.844352 23278 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20240502 14:13:41.844563 23278 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240502 14:13:41.847995 23278 ts_tablet_manager.cc:590] Registered 1 tablets
I20240502 14:13:41.848124 23278 ts_tablet_manager.cc:569] Time spent register tablets: real 0.004s user 0.000s sys 0.000s
I20240502 14:13:41.848984 23299 tablet_bootstrap.cc:492] T 41b085facc334a7da0c7dd937275009f P 92646999d5c64d1aae6f165593b7d14d: Bootstrap starting.
I20240502 14:13:41.872756 23299 log.cc:824] T 41b085facc334a7da0c7dd937275009f P 92646999d5c64d1aae6f165593b7d14d: Log is configured to *not* fsync() on all Append() calls
I20240502 14:13:41.877864 23299 tablet_bootstrap.cc:492] T 41b085facc334a7da0c7dd937275009f P 92646999d5c64d1aae6f165593b7d14d: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:13:41.878600 23299 tablet_bootstrap.cc:492] T 41b085facc334a7da0c7dd937275009f P 92646999d5c64d1aae6f165593b7d14d: Bootstrap complete.
I20240502 14:13:41.880218 23299 ts_tablet_manager.cc:1367] T 41b085facc334a7da0c7dd937275009f P 92646999d5c64d1aae6f165593b7d14d: Time spent bootstrapping tablet: real 0.032s user 0.016s sys 0.012s
I20240502 14:13:41.891793 23299 raft_consensus.cc:348] T 41b085facc334a7da0c7dd937275009f P 92646999d5c64d1aae6f165593b7d14d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92646999d5c64d1aae6f165593b7d14d" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 33817 } }
I20240502 14:13:41.892881 23299 raft_consensus.cc:729] T 41b085facc334a7da0c7dd937275009f P 92646999d5c64d1aae6f165593b7d14d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92646999d5c64d1aae6f165593b7d14d, State: Initialized, Role: FOLLOWER
I20240502 14:13:41.893802 23299 consensus_queue.cc:260] T 41b085facc334a7da0c7dd937275009f P 92646999d5c64d1aae6f165593b7d14d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92646999d5c64d1aae6f165593b7d14d" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 33817 } }
I20240502 14:13:41.894284 23299 raft_consensus.cc:388] T 41b085facc334a7da0c7dd937275009f P 92646999d5c64d1aae6f165593b7d14d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:41.894536 23299 raft_consensus.cc:482] T 41b085facc334a7da0c7dd937275009f P 92646999d5c64d1aae6f165593b7d14d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:41.894793 23299 raft_consensus.cc:3037] T 41b085facc334a7da0c7dd937275009f P 92646999d5c64d1aae6f165593b7d14d [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:13:41.894831 23278 rpc_server.cc:307] RPC server started. Bound to: 127.19.94.197:33817
I20240502 14:13:41.894944 23406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.94.197:33817 every 8 connection(s)
I20240502 14:13:41.896234 23278 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 14:13:41.898375 19835 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_zeJNl/build/asan/bin/kudu as pid 23278
I20240502 14:13:41.899017 19835 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_zeJNl/build/asan/bin/kudu
/tmp/dist-test-task_zeJNl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.94.198:41717
--local_ip_for_outbound_sockets=127.19.94.198
--tserver_master_addrs=127.19.94.254:40967
--webserver_port=44939
--webserver_interface=127.19.94.198
--builtin_ntp_servers=127.19.94.212:39595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:13:41.899823 23299 raft_consensus.cc:504] T 41b085facc334a7da0c7dd937275009f P 92646999d5c64d1aae6f165593b7d14d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92646999d5c64d1aae6f165593b7d14d" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 33817 } }
I20240502 14:13:41.900365 23299 leader_election.cc:302] T 41b085facc334a7da0c7dd937275009f P 92646999d5c64d1aae6f165593b7d14d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92646999d5c64d1aae6f165593b7d14d; no voters:
I20240502 14:13:41.901278 23409 raft_consensus.cc:2781] T 41b085facc334a7da0c7dd937275009f P 92646999d5c64d1aae6f165593b7d14d [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:13:41.901600 23299 leader_election.cc:288] T 41b085facc334a7da0c7dd937275009f P 92646999d5c64d1aae6f165593b7d14d [CANDIDATE]: Term 2 election: Requested vote from peers
I20240502 14:13:41.907881 23299 ts_tablet_manager.cc:1398] T 41b085facc334a7da0c7dd937275009f P 92646999d5c64d1aae6f165593b7d14d: Time spent starting tablet: real 0.027s user 0.016s sys 0.007s
I20240502 14:13:41.908915 23409 raft_consensus.cc:686] T 41b085facc334a7da0c7dd937275009f P 92646999d5c64d1aae6f165593b7d14d [term 2 LEADER]: Becoming Leader. State: Replica: 92646999d5c64d1aae6f165593b7d14d, State: Running, Role: LEADER
I20240502 14:13:41.909750 23409 consensus_queue.cc:237] T 41b085facc334a7da0c7dd937275009f P 92646999d5c64d1aae6f165593b7d14d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92646999d5c64d1aae6f165593b7d14d" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 33817 } }
I20240502 14:13:41.922663 23407 heartbeater.cc:347] Connected to a master server at 127.19.94.254:40967
I20240502 14:13:41.923046 23407 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:41.924026 23407 heartbeater.cc:511] Master 127.19.94.254:40967 requested a full tablet report, sending...
I20240502 14:13:41.927192 21722 ts_manager.cc:194] Re-registered known tserver with Master: 92646999d5c64d1aae6f165593b7d14d (127.19.94.197:33817)
I20240502 14:13:41.927739 21722 catalog_manager.cc:5484] T 41b085facc334a7da0c7dd937275009f P 92646999d5c64d1aae6f165593b7d14d reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "92646999d5c64d1aae6f165593b7d14d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92646999d5c64d1aae6f165593b7d14d" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 33817 } } }
I20240502 14:13:41.933560 21722 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.94.197:45235
I20240502 14:13:41.937088 23407 heartbeater.cc:503] Master 127.19.94.254:40967 was elected leader, sending a full tablet report...
I20240502 14:13:41.939857 23143 heartbeater.cc:503] Master 127.19.94.254:40967 was elected leader, sending a full tablet report...
W20240502 14:13:42.264113 23413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:42.264492 23413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:42.264608 23413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:42.272357 23413 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:13:42.272563 23413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:42.272750 23413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.94.198
I20240502 14:13:42.282137 23413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.94.212:39595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.94.198:41717
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.19.94.198
--webserver_port=44939
--tserver_master_addrs=127.19.94.254:40967
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.94.198
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:42.283617 23413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:42.285455 23413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:42.298077 23424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:42.298697 23425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:42.299610 23427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:42.300819 23413 server_base.cc:1034] running on GCE node
I20240502 14:13:42.301537 23413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:42.302706 23413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:42.303951 23413 hybrid_clock.cc:648] HybridClock initialized: now 1714659222303886 us; error 75 us; skew 500 ppm
I20240502 14:13:42.304302 23413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:42.308244 23413 webserver.cc:460] Webserver started at http://127.19.94.198:44939/ using document root <none> and password file <none>
I20240502 14:13:42.309162 23413 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:42.309309 23413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:42.315893 23413 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20240502 14:13:42.320044 23433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:42.321761 23413 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240502 14:13:42.321981 23413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "d53de47e3dbd4a37bb81b478ae4d1e0e"
format_stamp: "Formatted at 2024-05-02 14:13:30 on dist-test-slave-3j2v"
I20240502 14:13:42.322969 23413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:42.357877 23413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:42.358770 23413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:42.359100 23413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:42.360116 23413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:42.361847 23413 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:42.361933 23413 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:42.362078 23413 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:42.362140 23413 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:42.395992 23413 rpc_server.cc:307] RPC server started. Bound to: 127.19.94.198:41717
I20240502 14:13:42.396037 23545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.94.198:41717 every 8 connection(s)
I20240502 14:13:42.397516 23413 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 14:13:42.407954 19835 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_zeJNl/build/asan/bin/kudu as pid 23413
I20240502 14:13:42.408991 19835 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_zeJNl/build/asan/bin/kudu
/tmp/dist-test-task_zeJNl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.94.199:44599
--local_ip_for_outbound_sockets=127.19.94.199
--tserver_master_addrs=127.19.94.254:40967
--webserver_port=43897
--webserver_interface=127.19.94.199
--builtin_ntp_servers=127.19.94.212:39595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:13:42.413373 23546 heartbeater.cc:347] Connected to a master server at 127.19.94.254:40967
I20240502 14:13:42.413800 23546 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:42.414733 23546 heartbeater.cc:511] Master 127.19.94.254:40967 requested a full tablet report, sending...
I20240502 14:13:42.416539 21722 ts_manager.cc:194] Re-registered known tserver with Master: d53de47e3dbd4a37bb81b478ae4d1e0e (127.19.94.198:41717)
I20240502 14:13:42.417480 21722 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.94.198:38617
I20240502 14:13:42.432139 23275 heartbeater.cc:503] Master 127.19.94.254:40967 was elected leader, sending a full tablet report...
W20240502 14:13:42.781092 23549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:42.781423 23549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:42.781574 23549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:42.790056 23549 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:13:42.790218 23549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:42.790426 23549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.94.199
I20240502 14:13:42.801316 23549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.94.212:39595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.94.199:44599
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.19.94.199
--webserver_port=43897
--tserver_master_addrs=127.19.94.254:40967
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.94.199
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:42.803161 23549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:42.805330 23549 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:42.822065 23554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:42.822691 23557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:42.823446 23555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:42.824271 23549 server_base.cc:1034] running on GCE node
I20240502 14:13:42.825006 23549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:42.826153 23549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:42.827463 23549 hybrid_clock.cc:648] HybridClock initialized: now 1714659222827330 us; error 55 us; skew 500 ppm
I20240502 14:13:42.827966 23549 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:42.830826 23549 webserver.cc:460] Webserver started at http://127.19.94.199:43897/ using document root <none> and password file <none>
I20240502 14:13:42.831627 23549 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:42.831774 23549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:42.838389 23549 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20240502 14:13:42.841989 23563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:42.843683 23549 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20240502 14:13:42.843917 23549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "5ab00105ba8d4412bfb9d009c4e40352"
format_stamp: "Formatted at 2024-05-02 14:13:31 on dist-test-slave-3j2v"
I20240502 14:13:42.845016 23549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:42.865984 23549 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:42.866892 23549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:42.867216 23549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:42.868249 23549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:42.870110 23549 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:42.870201 23549 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:42.870349 23549 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:42.870407 23549 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:42.927284 23549 rpc_server.cc:307] RPC server started. Bound to: 127.19.94.199:44599
I20240502 14:13:42.927356 23675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.94.199:44599 every 8 connection(s)
I20240502 14:13:42.928936 23549 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 14:13:42.933116 19835 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_zeJNl/build/asan/bin/kudu as pid 23549
I20240502 14:13:42.945600 23676 heartbeater.cc:347] Connected to a master server at 127.19.94.254:40967
I20240502 14:13:42.945940 23676 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:42.946760 23676 heartbeater.cc:511] Master 127.19.94.254:40967 requested a full tablet report, sending...
I20240502 14:13:42.948379 21722 ts_manager.cc:194] Re-registered known tserver with Master: 5ab00105ba8d4412bfb9d009c4e40352 (127.19.94.199:44599)
I20240502 14:13:42.949677 21722 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.94.199:53605
I20240502 14:13:43.420562 23546 heartbeater.cc:503] Master 127.19.94.254:40967 was elected leader, sending a full tablet report...
I20240502 14:13:43.493388 21844 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:13:43.520707 23611 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:43.530527 23341 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:43.551273 21974 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:13:43.557020 23481 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:43.579111 23078 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:43.580045 23210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:43.802492 23611 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:43.803809 21844 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:13:43.825604 23341 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:43.828663 21974 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:13:43.840662 23481 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:43.858140 23210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:43.877665 23078 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:43.953069 23676 heartbeater.cc:503] Master 127.19.94.254:40967 was elected leader, sending a full tablet report...
I20240502 14:13:43.994844 21864 consensus_queue.cc:237] T 4cc5c4909e7343578a87c1b91ed3d756 P 0c1d247517e246c2828007ca0a6d5527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "5ab00105ba8d4412bfb9d009c4e40352" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 44599 } }
W20240502 14:13:44.005546 21798 consensus_peers.cc:485] T 4cc5c4909e7343578a87c1b91ed3d756 P 0c1d247517e246c2828007ca0a6d5527 -> Peer 5ab00105ba8d4412bfb9d009c4e40352 (127.19.94.199:44599): Couldn't send request to peer 5ab00105ba8d4412bfb9d009c4e40352. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4cc5c4909e7343578a87c1b91ed3d756. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:13:44.378552 23779 ts_tablet_manager.cc:902] T 4cc5c4909e7343578a87c1b91ed3d756 P 5ab00105ba8d4412bfb9d009c4e40352: Initiating tablet copy from peer 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193:38827)
I20240502 14:13:44.380584 23779 tablet_copy_client.cc:308] T 4cc5c4909e7343578a87c1b91ed3d756 P 5ab00105ba8d4412bfb9d009c4e40352: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.193:38827
I20240502 14:13:44.388736 21884 tablet_copy_service.cc:140] P 0c1d247517e246c2828007ca0a6d5527: Received BeginTabletCopySession request for tablet 4cc5c4909e7343578a87c1b91ed3d756 from peer 5ab00105ba8d4412bfb9d009c4e40352 ({username='slave'} at 127.19.94.199:37681)
I20240502 14:13:44.389029 21884 tablet_copy_service.cc:161] P 0c1d247517e246c2828007ca0a6d5527: Beginning new tablet copy session on tablet 4cc5c4909e7343578a87c1b91ed3d756 from peer 5ab00105ba8d4412bfb9d009c4e40352 at {username='slave'} at 127.19.94.199:37681: session id = 5ab00105ba8d4412bfb9d009c4e40352-4cc5c4909e7343578a87c1b91ed3d756
I20240502 14:13:44.391889 21884 tablet_copy_source_session.cc:205] T 4cc5c4909e7343578a87c1b91ed3d756 P 0c1d247517e246c2828007ca0a6d5527: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:13:44.396647 23779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cc5c4909e7343578a87c1b91ed3d756. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:44.410526 23779 tablet_copy_client.cc:783] T 4cc5c4909e7343578a87c1b91ed3d756 P 5ab00105ba8d4412bfb9d009c4e40352: tablet copy: Starting download of 0 data blocks...
I20240502 14:13:44.411201 23779 tablet_copy_client.cc:647] T 4cc5c4909e7343578a87c1b91ed3d756 P 5ab00105ba8d4412bfb9d009c4e40352: tablet copy: Starting download of 1 WAL segments...
I20240502 14:13:44.415318 23779 tablet_copy_client.cc:523] T 4cc5c4909e7343578a87c1b91ed3d756 P 5ab00105ba8d4412bfb9d009c4e40352: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:13:44.420097 23779 tablet_bootstrap.cc:492] T 4cc5c4909e7343578a87c1b91ed3d756 P 5ab00105ba8d4412bfb9d009c4e40352: Bootstrap starting.
I20240502 14:13:44.432313 23779 log.cc:824] T 4cc5c4909e7343578a87c1b91ed3d756 P 5ab00105ba8d4412bfb9d009c4e40352: Log is configured to *not* fsync() on all Append() calls
I20240502 14:13:44.469760 23779 tablet_bootstrap.cc:492] T 4cc5c4909e7343578a87c1b91ed3d756 P 5ab00105ba8d4412bfb9d009c4e40352: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:13:44.470374 23779 tablet_bootstrap.cc:492] T 4cc5c4909e7343578a87c1b91ed3d756 P 5ab00105ba8d4412bfb9d009c4e40352: Bootstrap complete.
I20240502 14:13:44.470857 23779 ts_tablet_manager.cc:1367] T 4cc5c4909e7343578a87c1b91ed3d756 P 5ab00105ba8d4412bfb9d009c4e40352: Time spent bootstrapping tablet: real 0.051s user 0.039s sys 0.012s
I20240502 14:13:44.475628 23779 raft_consensus.cc:348] T 4cc5c4909e7343578a87c1b91ed3d756 P 5ab00105ba8d4412bfb9d009c4e40352 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } }
I20240502 14:13:44.476619 23779 raft_consensus.cc:729] T 4cc5c4909e7343578a87c1b91ed3d756 P 5ab00105ba8d4412bfb9d009c4e40352 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ab00105ba8d4412bfb9d009c4e40352, State: Initialized, Role: FOLLOWER
I20240502 14:13:44.477109 23779 consensus_queue.cc:260] T 4cc5c4909e7343578a87c1b91ed3d756 P 5ab00105ba8d4412bfb9d009c4e40352 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "5ab00105ba8d4412bfb9d009c4e40352" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 44599 } }
I20240502 14:13:44.478544 23779 ts_tablet_manager.cc:1398] T 4cc5c4909e7343578a87c1b91ed3d756 P 5ab00105ba8d4412bfb9d009c4e40352: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20240502 14:13:44.479642 21884 tablet_copy_service.cc:342] P 0c1d247517e246c2828007ca0a6d5527: Request end of tablet copy session 5ab00105ba8d4412bfb9d009c4e40352-4cc5c4909e7343578a87c1b91ed3d756 received from {username='slave'} at 127.19.94.199:37681
I20240502 14:13:44.479840 21884 tablet_copy_service.cc:433] P 0c1d247517e246c2828007ca0a6d5527: ending tablet copy session 5ab00105ba8d4412bfb9d009c4e40352-4cc5c4909e7343578a87c1b91ed3d756 on tablet 4cc5c4909e7343578a87c1b91ed3d756 with peer 5ab00105ba8d4412bfb9d009c4e40352
I20240502 14:13:44.809652 23631 raft_consensus.cc:1203] T 4cc5c4909e7343578a87c1b91ed3d756 P 5ab00105ba8d4412bfb9d009c4e40352 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:13:44.811717 23787 raft_consensus.cc:2932] T 4cc5c4909e7343578a87c1b91ed3d756 P 0c1d247517e246c2828007ca0a6d5527 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5ab00105ba8d4412bfb9d009c4e40352 (127.19.94.199) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "5ab00105ba8d4412bfb9d009c4e40352" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 44599 } } }
I20240502 14:13:44.812898 23631 raft_consensus.cc:2932] T 4cc5c4909e7343578a87c1b91ed3d756 P 5ab00105ba8d4412bfb9d009c4e40352 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5ab00105ba8d4412bfb9d009c4e40352 (127.19.94.199) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "5ab00105ba8d4412bfb9d009c4e40352" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 44599 } } }
I20240502 14:13:44.818451 21718 catalog_manager.cc:5484] T 4cc5c4909e7343578a87c1b91ed3d756 P 0c1d247517e246c2828007ca0a6d5527 reported cstate change: config changed from index -1 to 4, VOTER 5ab00105ba8d4412bfb9d009c4e40352 (127.19.94.199) added. New cstate: current_term: 1 leader_uuid: "0c1d247517e246c2828007ca0a6d5527" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "5ab00105ba8d4412bfb9d009c4e40352" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 44599 } } }
I20240502 14:13:44.855052 21864 consensus_queue.cc:237] T 80edabe29f114dd2b3b958fc954e3c2b P 0c1d247517e246c2828007ca0a6d5527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "ff15cd84fa6640b8921bbf05dc65612e" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 39713 } }
W20240502 14:13:44.864537 21797 consensus_peers.cc:485] T 80edabe29f114dd2b3b958fc954e3c2b P 0c1d247517e246c2828007ca0a6d5527 -> Peer ff15cd84fa6640b8921bbf05dc65612e (127.19.94.195:39713): Couldn't send request to peer ff15cd84fa6640b8921bbf05dc65612e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80edabe29f114dd2b3b958fc954e3c2b. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:13:45.438899 23812 ts_tablet_manager.cc:902] T 80edabe29f114dd2b3b958fc954e3c2b P ff15cd84fa6640b8921bbf05dc65612e: Initiating tablet copy from peer 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193:38827)
I20240502 14:13:45.440888 23812 tablet_copy_client.cc:308] T 80edabe29f114dd2b3b958fc954e3c2b P ff15cd84fa6640b8921bbf05dc65612e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.193:38827
I20240502 14:13:45.449476 21884 tablet_copy_service.cc:140] P 0c1d247517e246c2828007ca0a6d5527: Received BeginTabletCopySession request for tablet 80edabe29f114dd2b3b958fc954e3c2b from peer ff15cd84fa6640b8921bbf05dc65612e ({username='slave'} at 127.19.94.195:58805)
I20240502 14:13:45.449718 21884 tablet_copy_service.cc:161] P 0c1d247517e246c2828007ca0a6d5527: Beginning new tablet copy session on tablet 80edabe29f114dd2b3b958fc954e3c2b from peer ff15cd84fa6640b8921bbf05dc65612e at {username='slave'} at 127.19.94.195:58805: session id = ff15cd84fa6640b8921bbf05dc65612e-80edabe29f114dd2b3b958fc954e3c2b
I20240502 14:13:45.452982 21884 tablet_copy_source_session.cc:205] T 80edabe29f114dd2b3b958fc954e3c2b P 0c1d247517e246c2828007ca0a6d5527: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:13:45.458340 23812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80edabe29f114dd2b3b958fc954e3c2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:45.470116 23812 tablet_copy_client.cc:783] T 80edabe29f114dd2b3b958fc954e3c2b P ff15cd84fa6640b8921bbf05dc65612e: tablet copy: Starting download of 0 data blocks...
I20240502 14:13:45.470522 23812 tablet_copy_client.cc:647] T 80edabe29f114dd2b3b958fc954e3c2b P ff15cd84fa6640b8921bbf05dc65612e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:13:45.473559 23812 tablet_copy_client.cc:523] T 80edabe29f114dd2b3b958fc954e3c2b P ff15cd84fa6640b8921bbf05dc65612e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:13:45.477371 23812 tablet_bootstrap.cc:492] T 80edabe29f114dd2b3b958fc954e3c2b P ff15cd84fa6640b8921bbf05dc65612e: Bootstrap starting.
I20240502 14:13:45.489883 23812 log.cc:824] T 80edabe29f114dd2b3b958fc954e3c2b P ff15cd84fa6640b8921bbf05dc65612e: Log is configured to *not* fsync() on all Append() calls
I20240502 14:13:45.530164 23812 tablet_bootstrap.cc:492] T 80edabe29f114dd2b3b958fc954e3c2b P ff15cd84fa6640b8921bbf05dc65612e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:13:45.530867 23812 tablet_bootstrap.cc:492] T 80edabe29f114dd2b3b958fc954e3c2b P ff15cd84fa6640b8921bbf05dc65612e: Bootstrap complete.
I20240502 14:13:45.531332 23812 ts_tablet_manager.cc:1367] T 80edabe29f114dd2b3b958fc954e3c2b P ff15cd84fa6640b8921bbf05dc65612e: Time spent bootstrapping tablet: real 0.054s user 0.040s sys 0.012s
I20240502 14:13:45.536705 23812 raft_consensus.cc:348] T 80edabe29f114dd2b3b958fc954e3c2b P ff15cd84fa6640b8921bbf05dc65612e [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } }
I20240502 14:13:45.537690 23812 raft_consensus.cc:729] T 80edabe29f114dd2b3b958fc954e3c2b P ff15cd84fa6640b8921bbf05dc65612e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff15cd84fa6640b8921bbf05dc65612e, State: Initialized, Role: FOLLOWER
I20240502 14:13:45.538182 23812 consensus_queue.cc:260] T 80edabe29f114dd2b3b958fc954e3c2b P ff15cd84fa6640b8921bbf05dc65612e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "ff15cd84fa6640b8921bbf05dc65612e" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 39713 } }
I20240502 14:13:45.540089 23812 ts_tablet_manager.cc:1398] T 80edabe29f114dd2b3b958fc954e3c2b P ff15cd84fa6640b8921bbf05dc65612e: Time spent starting tablet: real 0.009s user 0.010s sys 0.001s
I20240502 14:13:45.541360 21884 tablet_copy_service.cc:342] P 0c1d247517e246c2828007ca0a6d5527: Request end of tablet copy session ff15cd84fa6640b8921bbf05dc65612e-80edabe29f114dd2b3b958fc954e3c2b received from {username='slave'} at 127.19.94.195:58805
I20240502 14:13:45.541522 21884 tablet_copy_service.cc:433] P 0c1d247517e246c2828007ca0a6d5527: ending tablet copy session ff15cd84fa6640b8921bbf05dc65612e-80edabe29f114dd2b3b958fc954e3c2b on tablet 80edabe29f114dd2b3b958fc954e3c2b with peer ff15cd84fa6640b8921bbf05dc65612e
I20240502 14:13:45.994536 23098 raft_consensus.cc:1203] T 80edabe29f114dd2b3b958fc954e3c2b P ff15cd84fa6640b8921bbf05dc65612e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:13:45.996345 23821 raft_consensus.cc:2932] T 80edabe29f114dd2b3b958fc954e3c2b P 0c1d247517e246c2828007ca0a6d5527 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ff15cd84fa6640b8921bbf05dc65612e (127.19.94.195) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "ff15cd84fa6640b8921bbf05dc65612e" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 39713 } } }
I20240502 14:13:45.997223 23098 raft_consensus.cc:2932] T 80edabe29f114dd2b3b958fc954e3c2b P ff15cd84fa6640b8921bbf05dc65612e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ff15cd84fa6640b8921bbf05dc65612e (127.19.94.195) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "ff15cd84fa6640b8921bbf05dc65612e" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 39713 } } }
I20240502 14:13:45.999688 23821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:33346 (request call id 0) took 1145 ms. Trace:
I20240502 14:13:45.999961 23821 rpcz_store.cc:276] 0502 14:13:44.854381 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 14:13:44.854486 (+ 105us) service_pool.cc:225] Handling call
0502 14:13:44.856048 (+ 1562us) log.cc:842] Serialized 260 byte log entry
0502 14:13:45.999624 (+1143576us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":103,"raft.queue_time_us":49,"raft.run_cpu_time_us":266,"raft.run_wall_time_us":266,"thread_start_us":370,"threads_started":1,"wal-append.queue_time_us":531,"wal-append.run_cpu_time_us":672,"wal-append.run_wall_time_us":1000710}
I20240502 14:13:46.003541 21718 catalog_manager.cc:5484] T 80edabe29f114dd2b3b958fc954e3c2b P 0c1d247517e246c2828007ca0a6d5527 reported cstate change: config changed from index -1 to 4, VOTER ff15cd84fa6640b8921bbf05dc65612e (127.19.94.195) added. New cstate: current_term: 1 leader_uuid: "0c1d247517e246c2828007ca0a6d5527" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "ff15cd84fa6640b8921bbf05dc65612e" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 39713 } } }
I20240502 14:13:46.039013 21995 consensus_queue.cc:237] T 2ffa22a372c143bc87194122a407425c P 61c05b91821b450f9853ce940d18372b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "d53de47e3dbd4a37bb81b478ae4d1e0e" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 41717 } }
W20240502 14:13:46.049903 21928 consensus_peers.cc:485] T 2ffa22a372c143bc87194122a407425c P 61c05b91821b450f9853ce940d18372b -> Peer d53de47e3dbd4a37bb81b478ae4d1e0e (127.19.94.198:41717): Couldn't send request to peer d53de47e3dbd4a37bb81b478ae4d1e0e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2ffa22a372c143bc87194122a407425c. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:13:46.512450 23849 ts_tablet_manager.cc:902] T 2ffa22a372c143bc87194122a407425c P d53de47e3dbd4a37bb81b478ae4d1e0e: Initiating tablet copy from peer 61c05b91821b450f9853ce940d18372b (127.19.94.194:40163)
I20240502 14:13:46.514447 23849 tablet_copy_client.cc:308] T 2ffa22a372c143bc87194122a407425c P d53de47e3dbd4a37bb81b478ae4d1e0e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.194:40163
I20240502 14:13:46.522464 22015 tablet_copy_service.cc:140] P 61c05b91821b450f9853ce940d18372b: Received BeginTabletCopySession request for tablet 2ffa22a372c143bc87194122a407425c from peer d53de47e3dbd4a37bb81b478ae4d1e0e ({username='slave'} at 127.19.94.198:60879)
I20240502 14:13:46.522891 22015 tablet_copy_service.cc:161] P 61c05b91821b450f9853ce940d18372b: Beginning new tablet copy session on tablet 2ffa22a372c143bc87194122a407425c from peer d53de47e3dbd4a37bb81b478ae4d1e0e at {username='slave'} at 127.19.94.198:60879: session id = d53de47e3dbd4a37bb81b478ae4d1e0e-2ffa22a372c143bc87194122a407425c
I20240502 14:13:46.526005 22015 tablet_copy_source_session.cc:205] T 2ffa22a372c143bc87194122a407425c P 61c05b91821b450f9853ce940d18372b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:13:46.530896 23849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ffa22a372c143bc87194122a407425c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:46.542631 23849 tablet_copy_client.cc:783] T 2ffa22a372c143bc87194122a407425c P d53de47e3dbd4a37bb81b478ae4d1e0e: tablet copy: Starting download of 0 data blocks...
I20240502 14:13:46.543025 23849 tablet_copy_client.cc:647] T 2ffa22a372c143bc87194122a407425c P d53de47e3dbd4a37bb81b478ae4d1e0e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:13:46.545894 23849 tablet_copy_client.cc:523] T 2ffa22a372c143bc87194122a407425c P d53de47e3dbd4a37bb81b478ae4d1e0e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:13:46.549691 23849 tablet_bootstrap.cc:492] T 2ffa22a372c143bc87194122a407425c P d53de47e3dbd4a37bb81b478ae4d1e0e: Bootstrap starting.
I20240502 14:13:46.559430 23849 log.cc:824] T 2ffa22a372c143bc87194122a407425c P d53de47e3dbd4a37bb81b478ae4d1e0e: Log is configured to *not* fsync() on all Append() calls
I20240502 14:13:46.590366 23849 tablet_bootstrap.cc:492] T 2ffa22a372c143bc87194122a407425c P d53de47e3dbd4a37bb81b478ae4d1e0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:13:46.590972 23849 tablet_bootstrap.cc:492] T 2ffa22a372c143bc87194122a407425c P d53de47e3dbd4a37bb81b478ae4d1e0e: Bootstrap complete.
I20240502 14:13:46.591445 23849 ts_tablet_manager.cc:1367] T 2ffa22a372c143bc87194122a407425c P d53de47e3dbd4a37bb81b478ae4d1e0e: Time spent bootstrapping tablet: real 0.042s user 0.039s sys 0.000s
I20240502 14:13:46.596254 23849 raft_consensus.cc:348] T 2ffa22a372c143bc87194122a407425c P d53de47e3dbd4a37bb81b478ae4d1e0e [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } }
I20240502 14:13:46.597229 23849 raft_consensus.cc:729] T 2ffa22a372c143bc87194122a407425c P d53de47e3dbd4a37bb81b478ae4d1e0e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d53de47e3dbd4a37bb81b478ae4d1e0e, State: Initialized, Role: FOLLOWER
I20240502 14:13:46.597677 23849 consensus_queue.cc:260] T 2ffa22a372c143bc87194122a407425c P d53de47e3dbd4a37bb81b478ae4d1e0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "d53de47e3dbd4a37bb81b478ae4d1e0e" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 41717 } }
I20240502 14:13:46.599210 23849 ts_tablet_manager.cc:1398] T 2ffa22a372c143bc87194122a407425c P d53de47e3dbd4a37bb81b478ae4d1e0e: Time spent starting tablet: real 0.008s user 0.010s sys 0.000s
I20240502 14:13:46.600348 22015 tablet_copy_service.cc:342] P 61c05b91821b450f9853ce940d18372b: Request end of tablet copy session d53de47e3dbd4a37bb81b478ae4d1e0e-2ffa22a372c143bc87194122a407425c received from {username='slave'} at 127.19.94.198:60879
I20240502 14:13:46.600582 22015 tablet_copy_service.cc:433] P 61c05b91821b450f9853ce940d18372b: ending tablet copy session d53de47e3dbd4a37bb81b478ae4d1e0e-2ffa22a372c143bc87194122a407425c on tablet 2ffa22a372c143bc87194122a407425c with peer d53de47e3dbd4a37bb81b478ae4d1e0e
I20240502 14:13:46.906714 23501 raft_consensus.cc:1203] T 2ffa22a372c143bc87194122a407425c P d53de47e3dbd4a37bb81b478ae4d1e0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:13:46.908787 23857 raft_consensus.cc:2932] T 2ffa22a372c143bc87194122a407425c P 61c05b91821b450f9853ce940d18372b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d53de47e3dbd4a37bb81b478ae4d1e0e (127.19.94.198) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "d53de47e3dbd4a37bb81b478ae4d1e0e" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 41717 } } }
I20240502 14:13:46.909518 23501 raft_consensus.cc:2932] T 2ffa22a372c143bc87194122a407425c P d53de47e3dbd4a37bb81b478ae4d1e0e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d53de47e3dbd4a37bb81b478ae4d1e0e (127.19.94.198) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "d53de47e3dbd4a37bb81b478ae4d1e0e" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 41717 } } }
I20240502 14:13:46.915884 21716 catalog_manager.cc:5484] T 2ffa22a372c143bc87194122a407425c P 61c05b91821b450f9853ce940d18372b reported cstate change: config changed from index -1 to 4, VOTER d53de47e3dbd4a37bb81b478ae4d1e0e (127.19.94.198) added. New cstate: current_term: 1 leader_uuid: "61c05b91821b450f9853ce940d18372b" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "d53de47e3dbd4a37bb81b478ae4d1e0e" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 41717 } } }
I20240502 14:13:46.977175 21995 consensus_queue.cc:237] T 148d828a22194299a04c2e93003bf379 P 61c05b91821b450f9853ce940d18372b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "f8fbf60396bb443697ce265de0eb063e" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 41835 } }
W20240502 14:13:46.988590 21928 consensus_peers.cc:485] T 148d828a22194299a04c2e93003bf379 P 61c05b91821b450f9853ce940d18372b -> Peer f8fbf60396bb443697ce265de0eb063e (127.19.94.196:41835): Couldn't send request to peer f8fbf60396bb443697ce265de0eb063e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 148d828a22194299a04c2e93003bf379. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:13:47.440724 23881 ts_tablet_manager.cc:902] T 148d828a22194299a04c2e93003bf379 P f8fbf60396bb443697ce265de0eb063e: Initiating tablet copy from peer 61c05b91821b450f9853ce940d18372b (127.19.94.194:40163)
I20240502 14:13:47.442814 23881 tablet_copy_client.cc:308] T 148d828a22194299a04c2e93003bf379 P f8fbf60396bb443697ce265de0eb063e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.194:40163
I20240502 14:13:47.450791 22015 tablet_copy_service.cc:140] P 61c05b91821b450f9853ce940d18372b: Received BeginTabletCopySession request for tablet 148d828a22194299a04c2e93003bf379 from peer f8fbf60396bb443697ce265de0eb063e ({username='slave'} at 127.19.94.196:40999)
I20240502 14:13:47.451048 22015 tablet_copy_service.cc:161] P 61c05b91821b450f9853ce940d18372b: Beginning new tablet copy session on tablet 148d828a22194299a04c2e93003bf379 from peer f8fbf60396bb443697ce265de0eb063e at {username='slave'} at 127.19.94.196:40999: session id = f8fbf60396bb443697ce265de0eb063e-148d828a22194299a04c2e93003bf379
I20240502 14:13:47.453155 22015 tablet_copy_source_session.cc:205] T 148d828a22194299a04c2e93003bf379 P 61c05b91821b450f9853ce940d18372b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:13:47.457620 23881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 148d828a22194299a04c2e93003bf379. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:47.469542 23881 tablet_copy_client.cc:783] T 148d828a22194299a04c2e93003bf379 P f8fbf60396bb443697ce265de0eb063e: tablet copy: Starting download of 0 data blocks...
I20240502 14:13:47.469844 23881 tablet_copy_client.cc:647] T 148d828a22194299a04c2e93003bf379 P f8fbf60396bb443697ce265de0eb063e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:13:47.472146 23881 tablet_copy_client.cc:523] T 148d828a22194299a04c2e93003bf379 P f8fbf60396bb443697ce265de0eb063e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:13:47.475361 23881 tablet_bootstrap.cc:492] T 148d828a22194299a04c2e93003bf379 P f8fbf60396bb443697ce265de0eb063e: Bootstrap starting.
I20240502 14:13:47.486188 23881 log.cc:824] T 148d828a22194299a04c2e93003bf379 P f8fbf60396bb443697ce265de0eb063e: Log is configured to *not* fsync() on all Append() calls
I20240502 14:13:47.519909 23881 tablet_bootstrap.cc:492] T 148d828a22194299a04c2e93003bf379 P f8fbf60396bb443697ce265de0eb063e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:13:47.520617 23881 tablet_bootstrap.cc:492] T 148d828a22194299a04c2e93003bf379 P f8fbf60396bb443697ce265de0eb063e: Bootstrap complete.
I20240502 14:13:47.521072 23881 ts_tablet_manager.cc:1367] T 148d828a22194299a04c2e93003bf379 P f8fbf60396bb443697ce265de0eb063e: Time spent bootstrapping tablet: real 0.046s user 0.040s sys 0.007s
I20240502 14:13:47.527377 23881 raft_consensus.cc:348] T 148d828a22194299a04c2e93003bf379 P f8fbf60396bb443697ce265de0eb063e [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } }
I20240502 14:13:47.528705 23881 raft_consensus.cc:729] T 148d828a22194299a04c2e93003bf379 P f8fbf60396bb443697ce265de0eb063e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8fbf60396bb443697ce265de0eb063e, State: Initialized, Role: FOLLOWER
I20240502 14:13:47.529443 23881 consensus_queue.cc:260] T 148d828a22194299a04c2e93003bf379 P f8fbf60396bb443697ce265de0eb063e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "f8fbf60396bb443697ce265de0eb063e" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 41835 } }
I20240502 14:13:47.531495 23881 ts_tablet_manager.cc:1398] T 148d828a22194299a04c2e93003bf379 P f8fbf60396bb443697ce265de0eb063e: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20240502 14:13:47.532773 22015 tablet_copy_service.cc:342] P 61c05b91821b450f9853ce940d18372b: Request end of tablet copy session f8fbf60396bb443697ce265de0eb063e-148d828a22194299a04c2e93003bf379 received from {username='slave'} at 127.19.94.196:40999
I20240502 14:13:47.532948 22015 tablet_copy_service.cc:433] P 61c05b91821b450f9853ce940d18372b: ending tablet copy session f8fbf60396bb443697ce265de0eb063e-148d828a22194299a04c2e93003bf379 on tablet 148d828a22194299a04c2e93003bf379 with peer f8fbf60396bb443697ce265de0eb063e
I20240502 14:13:48.036335 23230 raft_consensus.cc:1203] T 148d828a22194299a04c2e93003bf379 P f8fbf60396bb443697ce265de0eb063e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:13:48.038496 23891 raft_consensus.cc:2932] T 148d828a22194299a04c2e93003bf379 P 61c05b91821b450f9853ce940d18372b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f8fbf60396bb443697ce265de0eb063e (127.19.94.196) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "f8fbf60396bb443697ce265de0eb063e" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 41835 } } }
I20240502 14:13:48.039525 23230 raft_consensus.cc:2932] T 148d828a22194299a04c2e93003bf379 P f8fbf60396bb443697ce265de0eb063e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f8fbf60396bb443697ce265de0eb063e (127.19.94.196) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "f8fbf60396bb443697ce265de0eb063e" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 41835 } } }
I20240502 14:13:48.041642 23891 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:46502 (request call id 0) took 1065 ms. Trace:
I20240502 14:13:48.041946 23891 rpcz_store.cc:276] 0502 14:13:46.976485 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 14:13:46.976560 (+ 75us) service_pool.cc:225] Handling call
0502 14:13:46.978050 (+ 1490us) log.cc:842] Serialized 260 byte log entry
0502 14:13:48.041588 (+1063538us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":92,"raft.queue_time_us":72,"raft.run_cpu_time_us":275,"raft.run_wall_time_us":275,"thread_start_us":289,"threads_started":1,"wal-append.queue_time_us":433,"wal-append.run_cpu_time_us":639,"wal-append.run_wall_time_us":1000714}
I20240502 14:13:48.044831 21716 catalog_manager.cc:5484] T 148d828a22194299a04c2e93003bf379 P 61c05b91821b450f9853ce940d18372b reported cstate change: config changed from index -1 to 4, VOTER f8fbf60396bb443697ce265de0eb063e (127.19.94.196) added. New cstate: current_term: 1 leader_uuid: "61c05b91821b450f9853ce940d18372b" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "f8fbf60396bb443697ce265de0eb063e" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 41835 } } }
I20240502 14:13:48.075856 21864 consensus_queue.cc:237] T f3119216db7d4c0ea2216744e6f21c5f P 0c1d247517e246c2828007ca0a6d5527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "92646999d5c64d1aae6f165593b7d14d" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 33817 } }
W20240502 14:13:48.084488 21797 consensus_peers.cc:485] T f3119216db7d4c0ea2216744e6f21c5f P 0c1d247517e246c2828007ca0a6d5527 -> Peer 92646999d5c64d1aae6f165593b7d14d (127.19.94.197:33817): Couldn't send request to peer 92646999d5c64d1aae6f165593b7d14d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3119216db7d4c0ea2216744e6f21c5f. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:13:48.656697 23918 ts_tablet_manager.cc:902] T f3119216db7d4c0ea2216744e6f21c5f P 92646999d5c64d1aae6f165593b7d14d: Initiating tablet copy from peer 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193:38827)
I20240502 14:13:48.658340 23918 tablet_copy_client.cc:308] T f3119216db7d4c0ea2216744e6f21c5f P 92646999d5c64d1aae6f165593b7d14d: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.193:38827
I20240502 14:13:48.665139 21884 tablet_copy_service.cc:140] P 0c1d247517e246c2828007ca0a6d5527: Received BeginTabletCopySession request for tablet f3119216db7d4c0ea2216744e6f21c5f from peer 92646999d5c64d1aae6f165593b7d14d ({username='slave'} at 127.19.94.197:36149)
I20240502 14:13:48.665350 21884 tablet_copy_service.cc:161] P 0c1d247517e246c2828007ca0a6d5527: Beginning new tablet copy session on tablet f3119216db7d4c0ea2216744e6f21c5f from peer 92646999d5c64d1aae6f165593b7d14d at {username='slave'} at 127.19.94.197:36149: session id = 92646999d5c64d1aae6f165593b7d14d-f3119216db7d4c0ea2216744e6f21c5f
I20240502 14:13:48.667379 21884 tablet_copy_source_session.cc:205] T f3119216db7d4c0ea2216744e6f21c5f P 0c1d247517e246c2828007ca0a6d5527: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:13:48.669842 23918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3119216db7d4c0ea2216744e6f21c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:48.675596 23918 tablet_copy_client.cc:783] T f3119216db7d4c0ea2216744e6f21c5f P 92646999d5c64d1aae6f165593b7d14d: tablet copy: Starting download of 0 data blocks...
I20240502 14:13:48.675899 23918 tablet_copy_client.cc:647] T f3119216db7d4c0ea2216744e6f21c5f P 92646999d5c64d1aae6f165593b7d14d: tablet copy: Starting download of 1 WAL segments...
I20240502 14:13:48.678105 23918 tablet_copy_client.cc:523] T f3119216db7d4c0ea2216744e6f21c5f P 92646999d5c64d1aae6f165593b7d14d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:13:48.681105 23918 tablet_bootstrap.cc:492] T f3119216db7d4c0ea2216744e6f21c5f P 92646999d5c64d1aae6f165593b7d14d: Bootstrap starting.
I20240502 14:13:48.719583 23918 tablet_bootstrap.cc:492] T f3119216db7d4c0ea2216744e6f21c5f P 92646999d5c64d1aae6f165593b7d14d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:13:48.720168 23918 tablet_bootstrap.cc:492] T f3119216db7d4c0ea2216744e6f21c5f P 92646999d5c64d1aae6f165593b7d14d: Bootstrap complete.
I20240502 14:13:48.720494 23918 ts_tablet_manager.cc:1367] T f3119216db7d4c0ea2216744e6f21c5f P 92646999d5c64d1aae6f165593b7d14d: Time spent bootstrapping tablet: real 0.039s user 0.036s sys 0.004s
I20240502 14:13:48.721421 23918 raft_consensus.cc:348] T f3119216db7d4c0ea2216744e6f21c5f P 92646999d5c64d1aae6f165593b7d14d [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } }
I20240502 14:13:48.721982 23918 raft_consensus.cc:729] T f3119216db7d4c0ea2216744e6f21c5f P 92646999d5c64d1aae6f165593b7d14d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92646999d5c64d1aae6f165593b7d14d, State: Initialized, Role: FOLLOWER
I20240502 14:13:48.722208 23918 consensus_queue.cc:260] T f3119216db7d4c0ea2216744e6f21c5f P 92646999d5c64d1aae6f165593b7d14d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "92646999d5c64d1aae6f165593b7d14d" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 33817 } }
I20240502 14:13:48.723001 23918 ts_tablet_manager.cc:1398] T f3119216db7d4c0ea2216744e6f21c5f P 92646999d5c64d1aae6f165593b7d14d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:13:48.723989 21884 tablet_copy_service.cc:342] P 0c1d247517e246c2828007ca0a6d5527: Request end of tablet copy session 92646999d5c64d1aae6f165593b7d14d-f3119216db7d4c0ea2216744e6f21c5f received from {username='slave'} at 127.19.94.197:36149
I20240502 14:13:48.724131 21884 tablet_copy_service.cc:433] P 0c1d247517e246c2828007ca0a6d5527: ending tablet copy session 92646999d5c64d1aae6f165593b7d14d-f3119216db7d4c0ea2216744e6f21c5f on tablet f3119216db7d4c0ea2216744e6f21c5f with peer 92646999d5c64d1aae6f165593b7d14d
I20240502 14:13:49.039407 23361 raft_consensus.cc:1203] T f3119216db7d4c0ea2216744e6f21c5f P 92646999d5c64d1aae6f165593b7d14d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:13:49.041242 23927 raft_consensus.cc:2932] T f3119216db7d4c0ea2216744e6f21c5f P 0c1d247517e246c2828007ca0a6d5527 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 92646999d5c64d1aae6f165593b7d14d (127.19.94.197) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "92646999d5c64d1aae6f165593b7d14d" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 33817 } } }
I20240502 14:13:49.041908 23361 raft_consensus.cc:2932] T f3119216db7d4c0ea2216744e6f21c5f P 92646999d5c64d1aae6f165593b7d14d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 92646999d5c64d1aae6f165593b7d14d (127.19.94.197) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "92646999d5c64d1aae6f165593b7d14d" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 33817 } } }
I20240502 14:13:49.048084 21715 catalog_manager.cc:5484] T f3119216db7d4c0ea2216744e6f21c5f P 92646999d5c64d1aae6f165593b7d14d reported cstate change: config changed from index -1 to 4, VOTER 92646999d5c64d1aae6f165593b7d14d (127.19.94.197) added. New cstate: current_term: 1 leader_uuid: "0c1d247517e246c2828007ca0a6d5527" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "92646999d5c64d1aae6f165593b7d14d" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 33817 } } }
I20240502 14:13:49.085507 21995 consensus_queue.cc:237] T 906a106c412944e4a049f1491999ad37 P 61c05b91821b450f9853ce940d18372b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "5ab00105ba8d4412bfb9d009c4e40352" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 44599 } }
W20240502 14:13:49.093699 21929 consensus_peers.cc:485] T 906a106c412944e4a049f1491999ad37 P 61c05b91821b450f9853ce940d18372b -> Peer 5ab00105ba8d4412bfb9d009c4e40352 (127.19.94.199:44599): Couldn't send request to peer 5ab00105ba8d4412bfb9d009c4e40352. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 906a106c412944e4a049f1491999ad37. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:13:49.465457 23954 ts_tablet_manager.cc:902] T 906a106c412944e4a049f1491999ad37 P 5ab00105ba8d4412bfb9d009c4e40352: Initiating tablet copy from peer 61c05b91821b450f9853ce940d18372b (127.19.94.194:40163)
I20240502 14:13:49.466645 23954 tablet_copy_client.cc:308] T 906a106c412944e4a049f1491999ad37 P 5ab00105ba8d4412bfb9d009c4e40352: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.194:40163
I20240502 14:13:49.472611 22015 tablet_copy_service.cc:140] P 61c05b91821b450f9853ce940d18372b: Received BeginTabletCopySession request for tablet 906a106c412944e4a049f1491999ad37 from peer 5ab00105ba8d4412bfb9d009c4e40352 ({username='slave'} at 127.19.94.199:52653)
I20240502 14:13:49.472774 22015 tablet_copy_service.cc:161] P 61c05b91821b450f9853ce940d18372b: Beginning new tablet copy session on tablet 906a106c412944e4a049f1491999ad37 from peer 5ab00105ba8d4412bfb9d009c4e40352 at {username='slave'} at 127.19.94.199:52653: session id = 5ab00105ba8d4412bfb9d009c4e40352-906a106c412944e4a049f1491999ad37
I20240502 14:13:49.474776 22015 tablet_copy_source_session.cc:205] T 906a106c412944e4a049f1491999ad37 P 61c05b91821b450f9853ce940d18372b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:13:49.476014 23954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 906a106c412944e4a049f1491999ad37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:49.480368 23954 tablet_copy_client.cc:783] T 906a106c412944e4a049f1491999ad37 P 5ab00105ba8d4412bfb9d009c4e40352: tablet copy: Starting download of 0 data blocks...
I20240502 14:13:49.480651 23954 tablet_copy_client.cc:647] T 906a106c412944e4a049f1491999ad37 P 5ab00105ba8d4412bfb9d009c4e40352: tablet copy: Starting download of 1 WAL segments...
I20240502 14:13:49.482352 23954 tablet_copy_client.cc:523] T 906a106c412944e4a049f1491999ad37 P 5ab00105ba8d4412bfb9d009c4e40352: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:13:49.484772 23954 tablet_bootstrap.cc:492] T 906a106c412944e4a049f1491999ad37 P 5ab00105ba8d4412bfb9d009c4e40352: Bootstrap starting.
I20240502 14:13:49.513820 23954 tablet_bootstrap.cc:492] T 906a106c412944e4a049f1491999ad37 P 5ab00105ba8d4412bfb9d009c4e40352: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:13:49.514223 23954 tablet_bootstrap.cc:492] T 906a106c412944e4a049f1491999ad37 P 5ab00105ba8d4412bfb9d009c4e40352: Bootstrap complete.
I20240502 14:13:49.514456 23954 ts_tablet_manager.cc:1367] T 906a106c412944e4a049f1491999ad37 P 5ab00105ba8d4412bfb9d009c4e40352: Time spent bootstrapping tablet: real 0.030s user 0.028s sys 0.000s
I20240502 14:13:49.515273 23954 raft_consensus.cc:348] T 906a106c412944e4a049f1491999ad37 P 5ab00105ba8d4412bfb9d009c4e40352 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } }
I20240502 14:13:49.515648 23954 raft_consensus.cc:729] T 906a106c412944e4a049f1491999ad37 P 5ab00105ba8d4412bfb9d009c4e40352 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ab00105ba8d4412bfb9d009c4e40352, State: Initialized, Role: FOLLOWER
I20240502 14:13:49.515799 23954 consensus_queue.cc:260] T 906a106c412944e4a049f1491999ad37 P 5ab00105ba8d4412bfb9d009c4e40352 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "5ab00105ba8d4412bfb9d009c4e40352" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 44599 } }
I20240502 14:13:49.516547 23954 ts_tablet_manager.cc:1398] T 906a106c412944e4a049f1491999ad37 P 5ab00105ba8d4412bfb9d009c4e40352: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 14:13:49.517462 22015 tablet_copy_service.cc:342] P 61c05b91821b450f9853ce940d18372b: Request end of tablet copy session 5ab00105ba8d4412bfb9d009c4e40352-906a106c412944e4a049f1491999ad37 received from {username='slave'} at 127.19.94.199:52653
I20240502 14:13:49.517642 22015 tablet_copy_service.cc:433] P 61c05b91821b450f9853ce940d18372b: ending tablet copy session 5ab00105ba8d4412bfb9d009c4e40352-906a106c412944e4a049f1491999ad37 on tablet 906a106c412944e4a049f1491999ad37 with peer 5ab00105ba8d4412bfb9d009c4e40352
I20240502 14:13:50.080008 23631 raft_consensus.cc:1203] T 906a106c412944e4a049f1491999ad37 P 5ab00105ba8d4412bfb9d009c4e40352 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:13:50.081879 23963 raft_consensus.cc:2932] T 906a106c412944e4a049f1491999ad37 P 61c05b91821b450f9853ce940d18372b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5ab00105ba8d4412bfb9d009c4e40352 (127.19.94.199) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "5ab00105ba8d4412bfb9d009c4e40352" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 44599 } } }
I20240502 14:13:50.082433 23631 raft_consensus.cc:2932] T 906a106c412944e4a049f1491999ad37 P 5ab00105ba8d4412bfb9d009c4e40352 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5ab00105ba8d4412bfb9d009c4e40352 (127.19.94.199) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "5ab00105ba8d4412bfb9d009c4e40352" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 44599 } } }
I20240502 14:13:50.087543 21716 catalog_manager.cc:5484] T 906a106c412944e4a049f1491999ad37 P 5ab00105ba8d4412bfb9d009c4e40352 reported cstate change: config changed from index -1 to 4, VOTER 5ab00105ba8d4412bfb9d009c4e40352 (127.19.94.199) added. New cstate: current_term: 1 leader_uuid: "61c05b91821b450f9853ce940d18372b" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "5ab00105ba8d4412bfb9d009c4e40352" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 44599 } } }
I20240502 14:13:50.121238 21864 consensus_queue.cc:237] T 8af0e06b069e424a9ace44e06f12a6d3 P 0c1d247517e246c2828007ca0a6d5527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "ff15cd84fa6640b8921bbf05dc65612e" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 39713 } }
W20240502 14:13:50.123765 21797 consensus_peers.cc:485] T 8af0e06b069e424a9ace44e06f12a6d3 P 0c1d247517e246c2828007ca0a6d5527 -> Peer ff15cd84fa6640b8921bbf05dc65612e (127.19.94.195:39713): Couldn't send request to peer ff15cd84fa6640b8921bbf05dc65612e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8af0e06b069e424a9ace44e06f12a6d3. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:13:50.635394 23987 ts_tablet_manager.cc:902] T 8af0e06b069e424a9ace44e06f12a6d3 P ff15cd84fa6640b8921bbf05dc65612e: Initiating tablet copy from peer 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193:38827)
I20240502 14:13:50.636601 23987 tablet_copy_client.cc:308] T 8af0e06b069e424a9ace44e06f12a6d3 P ff15cd84fa6640b8921bbf05dc65612e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.193:38827
I20240502 14:13:50.637455 21884 tablet_copy_service.cc:140] P 0c1d247517e246c2828007ca0a6d5527: Received BeginTabletCopySession request for tablet 8af0e06b069e424a9ace44e06f12a6d3 from peer ff15cd84fa6640b8921bbf05dc65612e ({username='slave'} at 127.19.94.195:58805)
I20240502 14:13:50.637647 21884 tablet_copy_service.cc:161] P 0c1d247517e246c2828007ca0a6d5527: Beginning new tablet copy session on tablet 8af0e06b069e424a9ace44e06f12a6d3 from peer ff15cd84fa6640b8921bbf05dc65612e at {username='slave'} at 127.19.94.195:58805: session id = ff15cd84fa6640b8921bbf05dc65612e-8af0e06b069e424a9ace44e06f12a6d3
I20240502 14:13:50.640527 21884 tablet_copy_source_session.cc:205] T 8af0e06b069e424a9ace44e06f12a6d3 P 0c1d247517e246c2828007ca0a6d5527: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:13:50.642149 23987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8af0e06b069e424a9ace44e06f12a6d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:50.648038 23987 tablet_copy_client.cc:783] T 8af0e06b069e424a9ace44e06f12a6d3 P ff15cd84fa6640b8921bbf05dc65612e: tablet copy: Starting download of 0 data blocks...
I20240502 14:13:50.648322 23987 tablet_copy_client.cc:647] T 8af0e06b069e424a9ace44e06f12a6d3 P ff15cd84fa6640b8921bbf05dc65612e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:13:50.650519 23987 tablet_copy_client.cc:523] T 8af0e06b069e424a9ace44e06f12a6d3 P ff15cd84fa6640b8921bbf05dc65612e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:13:50.654353 23987 tablet_bootstrap.cc:492] T 8af0e06b069e424a9ace44e06f12a6d3 P ff15cd84fa6640b8921bbf05dc65612e: Bootstrap starting.
I20240502 14:13:50.685184 23987 tablet_bootstrap.cc:492] T 8af0e06b069e424a9ace44e06f12a6d3 P ff15cd84fa6640b8921bbf05dc65612e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:13:50.685603 23987 tablet_bootstrap.cc:492] T 8af0e06b069e424a9ace44e06f12a6d3 P ff15cd84fa6640b8921bbf05dc65612e: Bootstrap complete.
I20240502 14:13:50.685904 23987 ts_tablet_manager.cc:1367] T 8af0e06b069e424a9ace44e06f12a6d3 P ff15cd84fa6640b8921bbf05dc65612e: Time spent bootstrapping tablet: real 0.032s user 0.027s sys 0.004s
I20240502 14:13:50.686800 23987 raft_consensus.cc:348] T 8af0e06b069e424a9ace44e06f12a6d3 P ff15cd84fa6640b8921bbf05dc65612e [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } }
I20240502 14:13:50.687178 23987 raft_consensus.cc:729] T 8af0e06b069e424a9ace44e06f12a6d3 P ff15cd84fa6640b8921bbf05dc65612e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff15cd84fa6640b8921bbf05dc65612e, State: Initialized, Role: FOLLOWER
I20240502 14:13:50.687338 23987 consensus_queue.cc:260] T 8af0e06b069e424a9ace44e06f12a6d3 P ff15cd84fa6640b8921bbf05dc65612e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "ff15cd84fa6640b8921bbf05dc65612e" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 39713 } }
I20240502 14:13:50.688030 23987 ts_tablet_manager.cc:1398] T 8af0e06b069e424a9ace44e06f12a6d3 P ff15cd84fa6640b8921bbf05dc65612e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:13:50.689065 21884 tablet_copy_service.cc:342] P 0c1d247517e246c2828007ca0a6d5527: Request end of tablet copy session ff15cd84fa6640b8921bbf05dc65612e-8af0e06b069e424a9ace44e06f12a6d3 received from {username='slave'} at 127.19.94.195:58805
I20240502 14:13:50.689218 21884 tablet_copy_service.cc:433] P 0c1d247517e246c2828007ca0a6d5527: ending tablet copy session ff15cd84fa6640b8921bbf05dc65612e-8af0e06b069e424a9ace44e06f12a6d3 on tablet 8af0e06b069e424a9ace44e06f12a6d3 with peer ff15cd84fa6640b8921bbf05dc65612e
I20240502 14:13:51.105288 23098 raft_consensus.cc:1203] T 8af0e06b069e424a9ace44e06f12a6d3 P ff15cd84fa6640b8921bbf05dc65612e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:13:51.107313 23994 raft_consensus.cc:2932] T 8af0e06b069e424a9ace44e06f12a6d3 P 0c1d247517e246c2828007ca0a6d5527 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ff15cd84fa6640b8921bbf05dc65612e (127.19.94.195) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "ff15cd84fa6640b8921bbf05dc65612e" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 39713 } } }
I20240502 14:13:51.108175 23098 raft_consensus.cc:2932] T 8af0e06b069e424a9ace44e06f12a6d3 P ff15cd84fa6640b8921bbf05dc65612e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ff15cd84fa6640b8921bbf05dc65612e (127.19.94.195) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "ff15cd84fa6640b8921bbf05dc65612e" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 39713 } } }
I20240502 14:13:51.112331 21716 catalog_manager.cc:5484] T 8af0e06b069e424a9ace44e06f12a6d3 P 0c1d247517e246c2828007ca0a6d5527 reported cstate change: config changed from index -1 to 4, VOTER ff15cd84fa6640b8921bbf05dc65612e (127.19.94.195) added. New cstate: current_term: 1 leader_uuid: "0c1d247517e246c2828007ca0a6d5527" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "ff15cd84fa6640b8921bbf05dc65612e" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 39713 } } }
I20240502 14:13:51.150264 21864 consensus_queue.cc:237] T 424a2170ba374ed1982e550e8f7ebeea P 0c1d247517e246c2828007ca0a6d5527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "d53de47e3dbd4a37bb81b478ae4d1e0e" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 41717 } }
W20240502 14:13:51.158878 21797 consensus_peers.cc:485] T 424a2170ba374ed1982e550e8f7ebeea P 0c1d247517e246c2828007ca0a6d5527 -> Peer d53de47e3dbd4a37bb81b478ae4d1e0e (127.19.94.198:41717): Couldn't send request to peer d53de47e3dbd4a37bb81b478ae4d1e0e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 424a2170ba374ed1982e550e8f7ebeea. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:13:51.581735 24020 ts_tablet_manager.cc:902] T 424a2170ba374ed1982e550e8f7ebeea P d53de47e3dbd4a37bb81b478ae4d1e0e: Initiating tablet copy from peer 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193:38827)
I20240502 14:13:51.583304 24020 tablet_copy_client.cc:308] T 424a2170ba374ed1982e550e8f7ebeea P d53de47e3dbd4a37bb81b478ae4d1e0e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.193:38827
I20240502 14:13:51.589691 21884 tablet_copy_service.cc:140] P 0c1d247517e246c2828007ca0a6d5527: Received BeginTabletCopySession request for tablet 424a2170ba374ed1982e550e8f7ebeea from peer d53de47e3dbd4a37bb81b478ae4d1e0e ({username='slave'} at 127.19.94.198:44461)
I20240502 14:13:51.589865 21884 tablet_copy_service.cc:161] P 0c1d247517e246c2828007ca0a6d5527: Beginning new tablet copy session on tablet 424a2170ba374ed1982e550e8f7ebeea from peer d53de47e3dbd4a37bb81b478ae4d1e0e at {username='slave'} at 127.19.94.198:44461: session id = d53de47e3dbd4a37bb81b478ae4d1e0e-424a2170ba374ed1982e550e8f7ebeea
I20240502 14:13:51.592010 21884 tablet_copy_source_session.cc:205] T 424a2170ba374ed1982e550e8f7ebeea P 0c1d247517e246c2828007ca0a6d5527: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:13:51.593291 24020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 424a2170ba374ed1982e550e8f7ebeea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:51.597683 24020 tablet_copy_client.cc:783] T 424a2170ba374ed1982e550e8f7ebeea P d53de47e3dbd4a37bb81b478ae4d1e0e: tablet copy: Starting download of 0 data blocks...
I20240502 14:13:51.597919 24020 tablet_copy_client.cc:647] T 424a2170ba374ed1982e550e8f7ebeea P d53de47e3dbd4a37bb81b478ae4d1e0e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:13:51.599973 24020 tablet_copy_client.cc:523] T 424a2170ba374ed1982e550e8f7ebeea P d53de47e3dbd4a37bb81b478ae4d1e0e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:13:51.602699 24020 tablet_bootstrap.cc:492] T 424a2170ba374ed1982e550e8f7ebeea P d53de47e3dbd4a37bb81b478ae4d1e0e: Bootstrap starting.
I20240502 14:13:51.638537 24020 tablet_bootstrap.cc:492] T 424a2170ba374ed1982e550e8f7ebeea P d53de47e3dbd4a37bb81b478ae4d1e0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:13:51.638984 24020 tablet_bootstrap.cc:492] T 424a2170ba374ed1982e550e8f7ebeea P d53de47e3dbd4a37bb81b478ae4d1e0e: Bootstrap complete.
I20240502 14:13:51.639266 24020 ts_tablet_manager.cc:1367] T 424a2170ba374ed1982e550e8f7ebeea P d53de47e3dbd4a37bb81b478ae4d1e0e: Time spent bootstrapping tablet: real 0.037s user 0.024s sys 0.012s
I20240502 14:13:51.640120 24020 raft_consensus.cc:348] T 424a2170ba374ed1982e550e8f7ebeea P d53de47e3dbd4a37bb81b478ae4d1e0e [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } }
I20240502 14:13:51.640571 24020 raft_consensus.cc:729] T 424a2170ba374ed1982e550e8f7ebeea P d53de47e3dbd4a37bb81b478ae4d1e0e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d53de47e3dbd4a37bb81b478ae4d1e0e, State: Initialized, Role: FOLLOWER
I20240502 14:13:51.640784 24020 consensus_queue.cc:260] T 424a2170ba374ed1982e550e8f7ebeea P d53de47e3dbd4a37bb81b478ae4d1e0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "d53de47e3dbd4a37bb81b478ae4d1e0e" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 41717 } }
I20240502 14:13:51.641583 24020 ts_tablet_manager.cc:1398] T 424a2170ba374ed1982e550e8f7ebeea P d53de47e3dbd4a37bb81b478ae4d1e0e: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240502 14:13:51.642766 21884 tablet_copy_service.cc:342] P 0c1d247517e246c2828007ca0a6d5527: Request end of tablet copy session d53de47e3dbd4a37bb81b478ae4d1e0e-424a2170ba374ed1982e550e8f7ebeea received from {username='slave'} at 127.19.94.198:44461
I20240502 14:13:51.642896 21884 tablet_copy_service.cc:433] P 0c1d247517e246c2828007ca0a6d5527: ending tablet copy session d53de47e3dbd4a37bb81b478ae4d1e0e-424a2170ba374ed1982e550e8f7ebeea on tablet 424a2170ba374ed1982e550e8f7ebeea with peer d53de47e3dbd4a37bb81b478ae4d1e0e
I20240502 14:13:52.112545 23501 raft_consensus.cc:1203] T 424a2170ba374ed1982e550e8f7ebeea P d53de47e3dbd4a37bb81b478ae4d1e0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:13:52.114169 24028 raft_consensus.cc:2932] T 424a2170ba374ed1982e550e8f7ebeea P 0c1d247517e246c2828007ca0a6d5527 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d53de47e3dbd4a37bb81b478ae4d1e0e (127.19.94.198) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "d53de47e3dbd4a37bb81b478ae4d1e0e" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 41717 } } }
I20240502 14:13:52.114938 23501 raft_consensus.cc:2932] T 424a2170ba374ed1982e550e8f7ebeea P d53de47e3dbd4a37bb81b478ae4d1e0e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d53de47e3dbd4a37bb81b478ae4d1e0e (127.19.94.198) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "d53de47e3dbd4a37bb81b478ae4d1e0e" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 41717 } } }
I20240502 14:13:52.122442 21715 catalog_manager.cc:5484] T 424a2170ba374ed1982e550e8f7ebeea P 0c1d247517e246c2828007ca0a6d5527 reported cstate change: config changed from index -1 to 4, VOTER d53de47e3dbd4a37bb81b478ae4d1e0e (127.19.94.198) added. New cstate: current_term: 1 leader_uuid: "0c1d247517e246c2828007ca0a6d5527" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "d53de47e3dbd4a37bb81b478ae4d1e0e" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 41717 } } }
I20240502 14:13:52.159922 21995 consensus_queue.cc:237] T 48dfb482c48d408aa359c0b948237fd6 P 61c05b91821b450f9853ce940d18372b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "f8fbf60396bb443697ce265de0eb063e" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 41835 } }
W20240502 14:13:52.162195 21928 consensus_peers.cc:485] T 48dfb482c48d408aa359c0b948237fd6 P 61c05b91821b450f9853ce940d18372b -> Peer f8fbf60396bb443697ce265de0eb063e (127.19.94.196:41835): Couldn't send request to peer f8fbf60396bb443697ce265de0eb063e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48dfb482c48d408aa359c0b948237fd6. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:13:52.583492 24052 ts_tablet_manager.cc:902] T 48dfb482c48d408aa359c0b948237fd6 P f8fbf60396bb443697ce265de0eb063e: Initiating tablet copy from peer 61c05b91821b450f9853ce940d18372b (127.19.94.194:40163)
I20240502 14:13:52.585050 24052 tablet_copy_client.cc:308] T 48dfb482c48d408aa359c0b948237fd6 P f8fbf60396bb443697ce265de0eb063e: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.194:40163
I20240502 14:13:52.586110 22015 tablet_copy_service.cc:140] P 61c05b91821b450f9853ce940d18372b: Received BeginTabletCopySession request for tablet 48dfb482c48d408aa359c0b948237fd6 from peer f8fbf60396bb443697ce265de0eb063e ({username='slave'} at 127.19.94.196:40999)
I20240502 14:13:52.586309 22015 tablet_copy_service.cc:161] P 61c05b91821b450f9853ce940d18372b: Beginning new tablet copy session on tablet 48dfb482c48d408aa359c0b948237fd6 from peer f8fbf60396bb443697ce265de0eb063e at {username='slave'} at 127.19.94.196:40999: session id = f8fbf60396bb443697ce265de0eb063e-48dfb482c48d408aa359c0b948237fd6
I20240502 14:13:52.588493 22015 tablet_copy_source_session.cc:205] T 48dfb482c48d408aa359c0b948237fd6 P 61c05b91821b450f9853ce940d18372b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:13:52.590070 24052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48dfb482c48d408aa359c0b948237fd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:52.595037 24052 tablet_copy_client.cc:783] T 48dfb482c48d408aa359c0b948237fd6 P f8fbf60396bb443697ce265de0eb063e: tablet copy: Starting download of 0 data blocks...
I20240502 14:13:52.595328 24052 tablet_copy_client.cc:647] T 48dfb482c48d408aa359c0b948237fd6 P f8fbf60396bb443697ce265de0eb063e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:13:52.597272 24052 tablet_copy_client.cc:523] T 48dfb482c48d408aa359c0b948237fd6 P f8fbf60396bb443697ce265de0eb063e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:13:52.600062 24052 tablet_bootstrap.cc:492] T 48dfb482c48d408aa359c0b948237fd6 P f8fbf60396bb443697ce265de0eb063e: Bootstrap starting.
I20240502 14:13:52.635547 24052 tablet_bootstrap.cc:492] T 48dfb482c48d408aa359c0b948237fd6 P f8fbf60396bb443697ce265de0eb063e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:13:52.636045 24052 tablet_bootstrap.cc:492] T 48dfb482c48d408aa359c0b948237fd6 P f8fbf60396bb443697ce265de0eb063e: Bootstrap complete.
I20240502 14:13:52.636370 24052 ts_tablet_manager.cc:1367] T 48dfb482c48d408aa359c0b948237fd6 P f8fbf60396bb443697ce265de0eb063e: Time spent bootstrapping tablet: real 0.036s user 0.027s sys 0.008s
I20240502 14:13:52.637516 24052 raft_consensus.cc:348] T 48dfb482c48d408aa359c0b948237fd6 P f8fbf60396bb443697ce265de0eb063e [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } }
I20240502 14:13:52.637987 24052 raft_consensus.cc:729] T 48dfb482c48d408aa359c0b948237fd6 P f8fbf60396bb443697ce265de0eb063e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8fbf60396bb443697ce265de0eb063e, State: Initialized, Role: FOLLOWER
I20240502 14:13:52.638300 24052 consensus_queue.cc:260] T 48dfb482c48d408aa359c0b948237fd6 P f8fbf60396bb443697ce265de0eb063e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "f8fbf60396bb443697ce265de0eb063e" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 41835 } }
I20240502 14:13:52.639230 24052 ts_tablet_manager.cc:1398] T 48dfb482c48d408aa359c0b948237fd6 P f8fbf60396bb443697ce265de0eb063e: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240502 14:13:52.640271 22015 tablet_copy_service.cc:342] P 61c05b91821b450f9853ce940d18372b: Request end of tablet copy session f8fbf60396bb443697ce265de0eb063e-48dfb482c48d408aa359c0b948237fd6 received from {username='slave'} at 127.19.94.196:40999
I20240502 14:13:52.640512 22015 tablet_copy_service.cc:433] P 61c05b91821b450f9853ce940d18372b: ending tablet copy session f8fbf60396bb443697ce265de0eb063e-48dfb482c48d408aa359c0b948237fd6 on tablet 48dfb482c48d408aa359c0b948237fd6 with peer f8fbf60396bb443697ce265de0eb063e
I20240502 14:13:53.013054 23230 raft_consensus.cc:1203] T 48dfb482c48d408aa359c0b948237fd6 P f8fbf60396bb443697ce265de0eb063e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:13:53.014629 24059 raft_consensus.cc:2932] T 48dfb482c48d408aa359c0b948237fd6 P 61c05b91821b450f9853ce940d18372b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f8fbf60396bb443697ce265de0eb063e (127.19.94.196) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "f8fbf60396bb443697ce265de0eb063e" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 41835 } } }
I20240502 14:13:53.015555 23230 raft_consensus.cc:2932] T 48dfb482c48d408aa359c0b948237fd6 P f8fbf60396bb443697ce265de0eb063e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f8fbf60396bb443697ce265de0eb063e (127.19.94.196) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "f8fbf60396bb443697ce265de0eb063e" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 41835 } } }
I20240502 14:13:53.020895 21716 catalog_manager.cc:5484] T 48dfb482c48d408aa359c0b948237fd6 P f8fbf60396bb443697ce265de0eb063e reported cstate change: config changed from index -1 to 4, VOTER f8fbf60396bb443697ce265de0eb063e (127.19.94.196) added. New cstate: current_term: 1 leader_uuid: "61c05b91821b450f9853ce940d18372b" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "f8fbf60396bb443697ce265de0eb063e" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 41835 } } }
I20240502 14:13:53.163180 23611 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:53.164379 21974 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:13:53.190024 23481 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:53.198235 21844 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:13:53.201514 23341 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:53.247426 23078 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:53.262255 23210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:53.318863 21995 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "48dfb482c48d408aa359c0b948237fd6"
dest_uuid: "61c05b91821b450f9853ce940d18372b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46558
I20240502 14:13:53.319135 21995 raft_consensus.cc:595] T 48dfb482c48d408aa359c0b948237fd6 P 61c05b91821b450f9853ce940d18372b [term 1 LEADER]: Received request to transfer leadership
I20240502 14:13:53.439673 23611 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:53.440204 21844 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:13:53.446859 21974 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:13:53.458221 23481 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:53.482399 23210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:53.489495 23341 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:53.497534 23078 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:53.586402 24156 raft_consensus.cc:979] T 48dfb482c48d408aa359c0b948237fd6 P 61c05b91821b450f9853ce940d18372b: : Instructing follower f8fbf60396bb443697ce265de0eb063e to start an election
I20240502 14:13:53.586606 24156 raft_consensus.cc:1067] T 48dfb482c48d408aa359c0b948237fd6 P 61c05b91821b450f9853ce940d18372b [term 1 LEADER]: Signalling peer f8fbf60396bb443697ce265de0eb063e to start an election
I20240502 14:13:53.587601 23230 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "48dfb482c48d408aa359c0b948237fd6"
dest_uuid: "f8fbf60396bb443697ce265de0eb063e"
from {username='slave'} at 127.19.94.194:44397
I20240502 14:13:53.587883 23230 raft_consensus.cc:482] T 48dfb482c48d408aa359c0b948237fd6 P f8fbf60396bb443697ce265de0eb063e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:13:53.588083 23230 raft_consensus.cc:3037] T 48dfb482c48d408aa359c0b948237fd6 P f8fbf60396bb443697ce265de0eb063e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:13:53.590427 23230 raft_consensus.cc:504] T 48dfb482c48d408aa359c0b948237fd6 P f8fbf60396bb443697ce265de0eb063e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "f8fbf60396bb443697ce265de0eb063e" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 41835 } }
I20240502 14:13:53.591867 23230 leader_election.cc:288] T 48dfb482c48d408aa359c0b948237fd6 P f8fbf60396bb443697ce265de0eb063e [CANDIDATE]: Term 2 election: Requested vote from peers 61c05b91821b450f9853ce940d18372b (127.19.94.194:40163)
I20240502 14:13:53.592602 21995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48dfb482c48d408aa359c0b948237fd6" candidate_uuid: "f8fbf60396bb443697ce265de0eb063e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "61c05b91821b450f9853ce940d18372b"
I20240502 14:13:53.592888 21995 raft_consensus.cc:3032] T 48dfb482c48d408aa359c0b948237fd6 P 61c05b91821b450f9853ce940d18372b [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:13:53.592999 21995 raft_consensus.cc:729] T 48dfb482c48d408aa359c0b948237fd6 P 61c05b91821b450f9853ce940d18372b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 61c05b91821b450f9853ce940d18372b, State: Running, Role: LEADER
I20240502 14:13:53.593281 21995 consensus_queue.cc:260] T 48dfb482c48d408aa359c0b948237fd6 P 61c05b91821b450f9853ce940d18372b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "f8fbf60396bb443697ce265de0eb063e" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 41835 } }
I20240502 14:13:53.593524 21995 raft_consensus.cc:3037] T 48dfb482c48d408aa359c0b948237fd6 P 61c05b91821b450f9853ce940d18372b [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:13:53.595518 21995 raft_consensus.cc:2454] T 48dfb482c48d408aa359c0b948237fd6 P 61c05b91821b450f9853ce940d18372b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8fbf60396bb443697ce265de0eb063e in term 2.
I20240502 14:13:53.596150 23164 leader_election.cc:302] T 48dfb482c48d408aa359c0b948237fd6 P f8fbf60396bb443697ce265de0eb063e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 61c05b91821b450f9853ce940d18372b, f8fbf60396bb443697ce265de0eb063e; no voters:
I20240502 14:13:53.596918 24157 raft_consensus.cc:2781] T 48dfb482c48d408aa359c0b948237fd6 P f8fbf60396bb443697ce265de0eb063e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:13:53.597441 24157 raft_consensus.cc:686] T 48dfb482c48d408aa359c0b948237fd6 P f8fbf60396bb443697ce265de0eb063e [term 2 LEADER]: Becoming Leader. State: Replica: f8fbf60396bb443697ce265de0eb063e, State: Running, Role: LEADER
I20240502 14:13:53.597925 24157 consensus_queue.cc:237] T 48dfb482c48d408aa359c0b948237fd6 P f8fbf60396bb443697ce265de0eb063e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "f8fbf60396bb443697ce265de0eb063e" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 41835 } }
I20240502 14:13:53.602718 21716 catalog_manager.cc:5484] T 48dfb482c48d408aa359c0b948237fd6 P f8fbf60396bb443697ce265de0eb063e reported cstate change: term changed from 1 to 2, leader changed from 61c05b91821b450f9853ce940d18372b (127.19.94.194) to f8fbf60396bb443697ce265de0eb063e (127.19.94.196). New cstate: current_term: 2 leader_uuid: "f8fbf60396bb443697ce265de0eb063e" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "f8fbf60396bb443697ce265de0eb063e" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 41835 } } }
I20240502 14:13:53.687194 21844 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:13:53.704074 23611 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:53.704862 23341 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:53.706121 23481 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:53.707016 23210 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:53.707823 21974 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:13:53.739710 23078 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:53.807415 21864 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "424a2170ba374ed1982e550e8f7ebeea"
dest_uuid: "0c1d247517e246c2828007ca0a6d5527"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33438
I20240502 14:13:53.807718 21864 raft_consensus.cc:595] T 424a2170ba374ed1982e550e8f7ebeea P 0c1d247517e246c2828007ca0a6d5527 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:13:53.925616 21974 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:13:53.935861 23481 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:53.943470 23341 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:53.957413 21844 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:13:53.957814 23611 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:53.996093 23210 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:53.996289 23078 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:54.120579 21995 raft_consensus.cc:1261] T 48dfb482c48d408aa359c0b948237fd6 P 61c05b91821b450f9853ce940d18372b [term 2 FOLLOWER]: Refusing update from remote peer f8fbf60396bb443697ce265de0eb063e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:13:54.121515 24235 consensus_queue.cc:1035] T 48dfb482c48d408aa359c0b948237fd6 P f8fbf60396bb443697ce265de0eb063e [LEADER]: Connected to new peer: Peer: permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:13:54.199990 23611 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:54.201274 21844 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:13:54.210482 21974 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:13:54.217751 23341 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:54.231421 23481 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:54.250591 23210 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:54.261049 23078 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:54.322238 24265 raft_consensus.cc:979] T 424a2170ba374ed1982e550e8f7ebeea P 0c1d247517e246c2828007ca0a6d5527: : Instructing follower d53de47e3dbd4a37bb81b478ae4d1e0e to start an election
I20240502 14:13:54.322451 23994 raft_consensus.cc:1067] T 424a2170ba374ed1982e550e8f7ebeea P 0c1d247517e246c2828007ca0a6d5527 [term 1 LEADER]: Signalling peer d53de47e3dbd4a37bb81b478ae4d1e0e to start an election
I20240502 14:13:54.323542 23501 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "424a2170ba374ed1982e550e8f7ebeea"
dest_uuid: "d53de47e3dbd4a37bb81b478ae4d1e0e"
from {username='slave'} at 127.19.94.193:45303
I20240502 14:13:54.323890 23501 raft_consensus.cc:482] T 424a2170ba374ed1982e550e8f7ebeea P d53de47e3dbd4a37bb81b478ae4d1e0e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:13:54.324100 23501 raft_consensus.cc:3037] T 424a2170ba374ed1982e550e8f7ebeea P d53de47e3dbd4a37bb81b478ae4d1e0e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:13:54.326716 23501 raft_consensus.cc:504] T 424a2170ba374ed1982e550e8f7ebeea P d53de47e3dbd4a37bb81b478ae4d1e0e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "d53de47e3dbd4a37bb81b478ae4d1e0e" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 41717 } }
I20240502 14:13:54.328174 23501 leader_election.cc:288] T 424a2170ba374ed1982e550e8f7ebeea P d53de47e3dbd4a37bb81b478ae4d1e0e [CANDIDATE]: Term 2 election: Requested vote from peers 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193:38827)
I20240502 14:13:54.329027 21864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "424a2170ba374ed1982e550e8f7ebeea" candidate_uuid: "d53de47e3dbd4a37bb81b478ae4d1e0e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0c1d247517e246c2828007ca0a6d5527"
I20240502 14:13:54.329358 21864 raft_consensus.cc:3032] T 424a2170ba374ed1982e550e8f7ebeea P 0c1d247517e246c2828007ca0a6d5527 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:13:54.329445 21864 raft_consensus.cc:729] T 424a2170ba374ed1982e550e8f7ebeea P 0c1d247517e246c2828007ca0a6d5527 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0c1d247517e246c2828007ca0a6d5527, State: Running, Role: LEADER
I20240502 14:13:54.329717 21864 consensus_queue.cc:260] T 424a2170ba374ed1982e550e8f7ebeea P 0c1d247517e246c2828007ca0a6d5527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "d53de47e3dbd4a37bb81b478ae4d1e0e" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 41717 } }
I20240502 14:13:54.330098 21864 raft_consensus.cc:3037] T 424a2170ba374ed1982e550e8f7ebeea P 0c1d247517e246c2828007ca0a6d5527 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:13:54.332176 21864 raft_consensus.cc:2454] T 424a2170ba374ed1982e550e8f7ebeea P 0c1d247517e246c2828007ca0a6d5527 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d53de47e3dbd4a37bb81b478ae4d1e0e in term 2.
I20240502 14:13:54.332823 23435 leader_election.cc:302] T 424a2170ba374ed1982e550e8f7ebeea P d53de47e3dbd4a37bb81b478ae4d1e0e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0c1d247517e246c2828007ca0a6d5527, d53de47e3dbd4a37bb81b478ae4d1e0e; no voters:
I20240502 14:13:54.333559 24271 raft_consensus.cc:2781] T 424a2170ba374ed1982e550e8f7ebeea P d53de47e3dbd4a37bb81b478ae4d1e0e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:13:54.335346 24271 raft_consensus.cc:686] T 424a2170ba374ed1982e550e8f7ebeea P d53de47e3dbd4a37bb81b478ae4d1e0e [term 2 LEADER]: Becoming Leader. State: Replica: d53de47e3dbd4a37bb81b478ae4d1e0e, State: Running, Role: LEADER
I20240502 14:13:54.335932 24271 consensus_queue.cc:237] T 424a2170ba374ed1982e550e8f7ebeea P d53de47e3dbd4a37bb81b478ae4d1e0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "d53de47e3dbd4a37bb81b478ae4d1e0e" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 41717 } }
I20240502 14:13:54.339332 21995 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2ffa22a372c143bc87194122a407425c"
dest_uuid: "61c05b91821b450f9853ce940d18372b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46612
I20240502 14:13:54.339643 21995 raft_consensus.cc:595] T 2ffa22a372c143bc87194122a407425c P 61c05b91821b450f9853ce940d18372b [term 1 LEADER]: Received request to transfer leadership
I20240502 14:13:54.341847 21717 catalog_manager.cc:5484] T 424a2170ba374ed1982e550e8f7ebeea P d53de47e3dbd4a37bb81b478ae4d1e0e reported cstate change: term changed from 1 to 2, leader changed from 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193) to d53de47e3dbd4a37bb81b478ae4d1e0e (127.19.94.198). New cstate: current_term: 2 leader_uuid: "d53de47e3dbd4a37bb81b478ae4d1e0e" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "d53de47e3dbd4a37bb81b478ae4d1e0e" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 41717 } } }
I20240502 14:13:54.459703 21844 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:13:54.462926 23611 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:54.466624 21974 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:13:54.482007 23341 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:54.483043 23210 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:54.483289 23481 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:54.486304 23078 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:54.690140 21844 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:13:54.700757 23341 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:54.715276 21974 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:13:54.717396 23210 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:54.718977 23611 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:54.719600 23481 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:54.721251 21864 raft_consensus.cc:1261] T 424a2170ba374ed1982e550e8f7ebeea P 0c1d247517e246c2828007ca0a6d5527 [term 2 FOLLOWER]: Refusing update from remote peer d53de47e3dbd4a37bb81b478ae4d1e0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:13:54.721709 23078 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:54.722430 24271 consensus_queue.cc:1035] T 424a2170ba374ed1982e550e8f7ebeea P d53de47e3dbd4a37bb81b478ae4d1e0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:13:54.773609 24350 raft_consensus.cc:979] T 2ffa22a372c143bc87194122a407425c P 61c05b91821b450f9853ce940d18372b: : Instructing follower d53de47e3dbd4a37bb81b478ae4d1e0e to start an election
I20240502 14:13:54.781868 24156 raft_consensus.cc:1067] T 2ffa22a372c143bc87194122a407425c P 61c05b91821b450f9853ce940d18372b [term 1 LEADER]: Signalling peer d53de47e3dbd4a37bb81b478ae4d1e0e to start an election
I20240502 14:13:54.783094 23501 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2ffa22a372c143bc87194122a407425c"
dest_uuid: "d53de47e3dbd4a37bb81b478ae4d1e0e"
from {username='slave'} at 127.19.94.194:59917
I20240502 14:13:54.783651 23501 raft_consensus.cc:482] T 2ffa22a372c143bc87194122a407425c P d53de47e3dbd4a37bb81b478ae4d1e0e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:13:54.783922 23501 raft_consensus.cc:3037] T 2ffa22a372c143bc87194122a407425c P d53de47e3dbd4a37bb81b478ae4d1e0e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:13:54.787747 23501 raft_consensus.cc:504] T 2ffa22a372c143bc87194122a407425c P d53de47e3dbd4a37bb81b478ae4d1e0e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "d53de47e3dbd4a37bb81b478ae4d1e0e" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 41717 } }
I20240502 14:13:54.789034 23501 leader_election.cc:288] T 2ffa22a372c143bc87194122a407425c P d53de47e3dbd4a37bb81b478ae4d1e0e [CANDIDATE]: Term 2 election: Requested vote from peers 61c05b91821b450f9853ce940d18372b (127.19.94.194:40163)
I20240502 14:13:54.789451 21995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ffa22a372c143bc87194122a407425c" candidate_uuid: "d53de47e3dbd4a37bb81b478ae4d1e0e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "61c05b91821b450f9853ce940d18372b"
I20240502 14:13:54.789755 21995 raft_consensus.cc:3032] T 2ffa22a372c143bc87194122a407425c P 61c05b91821b450f9853ce940d18372b [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:13:54.789834 21995 raft_consensus.cc:729] T 2ffa22a372c143bc87194122a407425c P 61c05b91821b450f9853ce940d18372b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 61c05b91821b450f9853ce940d18372b, State: Running, Role: LEADER
I20240502 14:13:54.790140 21995 consensus_queue.cc:260] T 2ffa22a372c143bc87194122a407425c P 61c05b91821b450f9853ce940d18372b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "d53de47e3dbd4a37bb81b478ae4d1e0e" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 41717 } }
I20240502 14:13:54.790622 21995 raft_consensus.cc:3037] T 2ffa22a372c143bc87194122a407425c P 61c05b91821b450f9853ce940d18372b [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:13:54.794037 21995 raft_consensus.cc:2454] T 2ffa22a372c143bc87194122a407425c P 61c05b91821b450f9853ce940d18372b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d53de47e3dbd4a37bb81b478ae4d1e0e in term 2.
I20240502 14:13:54.794822 23436 leader_election.cc:302] T 2ffa22a372c143bc87194122a407425c P d53de47e3dbd4a37bb81b478ae4d1e0e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 61c05b91821b450f9853ce940d18372b, d53de47e3dbd4a37bb81b478ae4d1e0e; no voters:
I20240502 14:13:54.795241 24271 raft_consensus.cc:2781] T 2ffa22a372c143bc87194122a407425c P d53de47e3dbd4a37bb81b478ae4d1e0e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:13:54.795411 24271 raft_consensus.cc:686] T 2ffa22a372c143bc87194122a407425c P d53de47e3dbd4a37bb81b478ae4d1e0e [term 2 LEADER]: Becoming Leader. State: Replica: d53de47e3dbd4a37bb81b478ae4d1e0e, State: Running, Role: LEADER
I20240502 14:13:54.795742 24271 consensus_queue.cc:237] T 2ffa22a372c143bc87194122a407425c P d53de47e3dbd4a37bb81b478ae4d1e0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "d53de47e3dbd4a37bb81b478ae4d1e0e" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 41717 } }
I20240502 14:13:54.812654 21718 catalog_manager.cc:5484] T 2ffa22a372c143bc87194122a407425c P d53de47e3dbd4a37bb81b478ae4d1e0e reported cstate change: term changed from 1 to 2, leader changed from 61c05b91821b450f9853ce940d18372b (127.19.94.194) to d53de47e3dbd4a37bb81b478ae4d1e0e (127.19.94.198). New cstate: current_term: 2 leader_uuid: "d53de47e3dbd4a37bb81b478ae4d1e0e" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "d53de47e3dbd4a37bb81b478ae4d1e0e" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 41717 } } }
I20240502 14:13:54.850056 21864 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4cc5c4909e7343578a87c1b91ed3d756"
dest_uuid: "0c1d247517e246c2828007ca0a6d5527"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55154
I20240502 14:13:54.850289 21864 raft_consensus.cc:595] T 4cc5c4909e7343578a87c1b91ed3d756 P 0c1d247517e246c2828007ca0a6d5527 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:13:54.963517 23611 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:54.964093 21844 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:13:54.986323 21974 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:13:54.991837 23341 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:54.999575 23481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:55.036123 23210 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:55.045965 23078 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:55.097698 24388 raft_consensus.cc:979] T 4cc5c4909e7343578a87c1b91ed3d756 P 0c1d247517e246c2828007ca0a6d5527: : Instructing follower 5ab00105ba8d4412bfb9d009c4e40352 to start an election
I20240502 14:13:55.098040 23994 raft_consensus.cc:1067] T 4cc5c4909e7343578a87c1b91ed3d756 P 0c1d247517e246c2828007ca0a6d5527 [term 1 LEADER]: Signalling peer 5ab00105ba8d4412bfb9d009c4e40352 to start an election
I20240502 14:13:55.098950 23631 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4cc5c4909e7343578a87c1b91ed3d756"
dest_uuid: "5ab00105ba8d4412bfb9d009c4e40352"
from {username='slave'} at 127.19.94.193:58581
I20240502 14:13:55.099251 23631 raft_consensus.cc:482] T 4cc5c4909e7343578a87c1b91ed3d756 P 5ab00105ba8d4412bfb9d009c4e40352 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:13:55.099501 23631 raft_consensus.cc:3037] T 4cc5c4909e7343578a87c1b91ed3d756 P 5ab00105ba8d4412bfb9d009c4e40352 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:13:55.101864 23631 raft_consensus.cc:504] T 4cc5c4909e7343578a87c1b91ed3d756 P 5ab00105ba8d4412bfb9d009c4e40352 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "5ab00105ba8d4412bfb9d009c4e40352" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 44599 } }
I20240502 14:13:55.103428 23631 leader_election.cc:288] T 4cc5c4909e7343578a87c1b91ed3d756 P 5ab00105ba8d4412bfb9d009c4e40352 [CANDIDATE]: Term 2 election: Requested vote from peers 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193:38827)
I20240502 14:13:55.104897 21864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cc5c4909e7343578a87c1b91ed3d756" candidate_uuid: "5ab00105ba8d4412bfb9d009c4e40352" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0c1d247517e246c2828007ca0a6d5527"
I20240502 14:13:55.105167 21864 raft_consensus.cc:3032] T 4cc5c4909e7343578a87c1b91ed3d756 P 0c1d247517e246c2828007ca0a6d5527 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:13:55.105279 21864 raft_consensus.cc:729] T 4cc5c4909e7343578a87c1b91ed3d756 P 0c1d247517e246c2828007ca0a6d5527 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0c1d247517e246c2828007ca0a6d5527, State: Running, Role: LEADER
I20240502 14:13:55.105566 21864 consensus_queue.cc:260] T 4cc5c4909e7343578a87c1b91ed3d756 P 0c1d247517e246c2828007ca0a6d5527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "5ab00105ba8d4412bfb9d009c4e40352" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 44599 } }
I20240502 14:13:55.105909 21864 raft_consensus.cc:3037] T 4cc5c4909e7343578a87c1b91ed3d756 P 0c1d247517e246c2828007ca0a6d5527 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:13:55.108443 21864 raft_consensus.cc:2454] T 4cc5c4909e7343578a87c1b91ed3d756 P 0c1d247517e246c2828007ca0a6d5527 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ab00105ba8d4412bfb9d009c4e40352 in term 2.
I20240502 14:13:55.109367 23565 leader_election.cc:302] T 4cc5c4909e7343578a87c1b91ed3d756 P 5ab00105ba8d4412bfb9d009c4e40352 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0c1d247517e246c2828007ca0a6d5527, 5ab00105ba8d4412bfb9d009c4e40352; no voters:
I20240502 14:13:55.110633 24394 raft_consensus.cc:2781] T 4cc5c4909e7343578a87c1b91ed3d756 P 5ab00105ba8d4412bfb9d009c4e40352 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:13:55.111385 24394 raft_consensus.cc:686] T 4cc5c4909e7343578a87c1b91ed3d756 P 5ab00105ba8d4412bfb9d009c4e40352 [term 2 LEADER]: Becoming Leader. State: Replica: 5ab00105ba8d4412bfb9d009c4e40352, State: Running, Role: LEADER
I20240502 14:13:55.112320 24394 consensus_queue.cc:237] T 4cc5c4909e7343578a87c1b91ed3d756 P 5ab00105ba8d4412bfb9d009c4e40352 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "5ab00105ba8d4412bfb9d009c4e40352" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 44599 } }
I20240502 14:13:55.117555 21717 catalog_manager.cc:5484] T 4cc5c4909e7343578a87c1b91ed3d756 P 5ab00105ba8d4412bfb9d009c4e40352 reported cstate change: term changed from 1 to 2, leader changed from 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193) to 5ab00105ba8d4412bfb9d009c4e40352 (127.19.94.199). New cstate: current_term: 2 leader_uuid: "5ab00105ba8d4412bfb9d009c4e40352" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "5ab00105ba8d4412bfb9d009c4e40352" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 44599 } } }
I20240502 14:13:55.224704 21844 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:13:55.231765 23341 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:55.232890 21974 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:13:55.256108 23481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:55.258468 23210 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:55.277058 23611 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:55.289939 23078 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:55.362747 21995 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "148d828a22194299a04c2e93003bf379"
dest_uuid: "61c05b91821b450f9853ce940d18372b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58624
I20240502 14:13:55.362953 21995 raft_consensus.cc:595] T 148d828a22194299a04c2e93003bf379 P 61c05b91821b450f9853ce940d18372b [term 1 LEADER]: Received request to transfer leadership
I20240502 14:13:55.369372 24429 raft_consensus.cc:979] T 148d828a22194299a04c2e93003bf379 P 61c05b91821b450f9853ce940d18372b: : Instructing follower f8fbf60396bb443697ce265de0eb063e to start an election
I20240502 14:13:55.369540 24429 raft_consensus.cc:1067] T 148d828a22194299a04c2e93003bf379 P 61c05b91821b450f9853ce940d18372b [term 1 LEADER]: Signalling peer f8fbf60396bb443697ce265de0eb063e to start an election
I20240502 14:13:55.370168 23230 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "148d828a22194299a04c2e93003bf379"
dest_uuid: "f8fbf60396bb443697ce265de0eb063e"
from {username='slave'} at 127.19.94.194:44397
I20240502 14:13:55.370375 23230 raft_consensus.cc:482] T 148d828a22194299a04c2e93003bf379 P f8fbf60396bb443697ce265de0eb063e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:13:55.370481 23230 raft_consensus.cc:3037] T 148d828a22194299a04c2e93003bf379 P f8fbf60396bb443697ce265de0eb063e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:13:55.373384 23230 raft_consensus.cc:504] T 148d828a22194299a04c2e93003bf379 P f8fbf60396bb443697ce265de0eb063e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "f8fbf60396bb443697ce265de0eb063e" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 41835 } }
I20240502 14:13:55.374361 23230 leader_election.cc:288] T 148d828a22194299a04c2e93003bf379 P f8fbf60396bb443697ce265de0eb063e [CANDIDATE]: Term 2 election: Requested vote from peers 61c05b91821b450f9853ce940d18372b (127.19.94.194:40163)
I20240502 14:13:55.374480 21995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "148d828a22194299a04c2e93003bf379" candidate_uuid: "f8fbf60396bb443697ce265de0eb063e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "61c05b91821b450f9853ce940d18372b"
I20240502 14:13:55.374719 21995 raft_consensus.cc:3032] T 148d828a22194299a04c2e93003bf379 P 61c05b91821b450f9853ce940d18372b [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:13:55.374811 21995 raft_consensus.cc:729] T 148d828a22194299a04c2e93003bf379 P 61c05b91821b450f9853ce940d18372b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 61c05b91821b450f9853ce940d18372b, State: Running, Role: LEADER
I20240502 14:13:55.375082 21995 consensus_queue.cc:260] T 148d828a22194299a04c2e93003bf379 P 61c05b91821b450f9853ce940d18372b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "f8fbf60396bb443697ce265de0eb063e" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 41835 } }
I20240502 14:13:55.375313 21995 raft_consensus.cc:3037] T 148d828a22194299a04c2e93003bf379 P 61c05b91821b450f9853ce940d18372b [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:13:55.377163 21995 raft_consensus.cc:2454] T 148d828a22194299a04c2e93003bf379 P 61c05b91821b450f9853ce940d18372b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8fbf60396bb443697ce265de0eb063e in term 2.
I20240502 14:13:55.377604 23164 leader_election.cc:302] T 148d828a22194299a04c2e93003bf379 P f8fbf60396bb443697ce265de0eb063e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 61c05b91821b450f9853ce940d18372b, f8fbf60396bb443697ce265de0eb063e; no voters:
I20240502 14:13:55.378008 24335 raft_consensus.cc:2781] T 148d828a22194299a04c2e93003bf379 P f8fbf60396bb443697ce265de0eb063e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:13:55.378705 24335 raft_consensus.cc:686] T 148d828a22194299a04c2e93003bf379 P f8fbf60396bb443697ce265de0eb063e [term 2 LEADER]: Becoming Leader. State: Replica: f8fbf60396bb443697ce265de0eb063e, State: Running, Role: LEADER
I20240502 14:13:55.379138 24335 consensus_queue.cc:237] T 148d828a22194299a04c2e93003bf379 P f8fbf60396bb443697ce265de0eb063e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "f8fbf60396bb443697ce265de0eb063e" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 41835 } }
I20240502 14:13:55.382201 21717 catalog_manager.cc:5484] T 148d828a22194299a04c2e93003bf379 P f8fbf60396bb443697ce265de0eb063e reported cstate change: term changed from 1 to 2, leader changed from 61c05b91821b450f9853ce940d18372b (127.19.94.194) to f8fbf60396bb443697ce265de0eb063e (127.19.94.196). New cstate: current_term: 2 leader_uuid: "f8fbf60396bb443697ce265de0eb063e" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "f8fbf60396bb443697ce265de0eb063e" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 41835 } } }
I20240502 14:13:55.416345 21995 raft_consensus.cc:1261] T 2ffa22a372c143bc87194122a407425c P 61c05b91821b450f9853ce940d18372b [term 2 FOLLOWER]: Refusing update from remote peer d53de47e3dbd4a37bb81b478ae4d1e0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:13:55.417143 24271 consensus_queue.cc:1035] T 2ffa22a372c143bc87194122a407425c P d53de47e3dbd4a37bb81b478ae4d1e0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:13:55.483805 21844 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:13:55.500756 23611 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:55.526027 21974 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:13:55.530427 23210 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:55.531085 23341 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:55.540083 21864 raft_consensus.cc:1261] T 4cc5c4909e7343578a87c1b91ed3d756 P 0c1d247517e246c2828007ca0a6d5527 [term 2 FOLLOWER]: Refusing update from remote peer 5ab00105ba8d4412bfb9d009c4e40352: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:13:55.540633 23078 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:55.540827 24394 consensus_queue.cc:1035] T 4cc5c4909e7343578a87c1b91ed3d756 P 5ab00105ba8d4412bfb9d009c4e40352 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:13:55.577478 23481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:55.736146 21974 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:13:55.750945 21844 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:13:55.764895 23611 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:55.765463 23210 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:55.765686 23481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:55.796408 23341 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:55.810551 21995 raft_consensus.cc:1261] T 148d828a22194299a04c2e93003bf379 P 61c05b91821b450f9853ce940d18372b [term 2 FOLLOWER]: Refusing update from remote peer f8fbf60396bb443697ce265de0eb063e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:13:55.811587 24335 consensus_queue.cc:1035] T 148d828a22194299a04c2e93003bf379 P f8fbf60396bb443697ce265de0eb063e [LEADER]: Connected to new peer: Peer: permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:13:55.847358 23078 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:55.895478 21864 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "80edabe29f114dd2b3b958fc954e3c2b"
dest_uuid: "0c1d247517e246c2828007ca0a6d5527"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55214
I20240502 14:13:55.895710 21864 raft_consensus.cc:595] T 80edabe29f114dd2b3b958fc954e3c2b P 0c1d247517e246c2828007ca0a6d5527 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:13:55.940338 24527 raft_consensus.cc:979] T 80edabe29f114dd2b3b958fc954e3c2b P 0c1d247517e246c2828007ca0a6d5527: : Instructing follower ff15cd84fa6640b8921bbf05dc65612e to start an election
I20240502 14:13:55.940586 24527 raft_consensus.cc:1067] T 80edabe29f114dd2b3b958fc954e3c2b P 0c1d247517e246c2828007ca0a6d5527 [term 1 LEADER]: Signalling peer ff15cd84fa6640b8921bbf05dc65612e to start an election
I20240502 14:13:55.941439 23098 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "80edabe29f114dd2b3b958fc954e3c2b"
dest_uuid: "ff15cd84fa6640b8921bbf05dc65612e"
from {username='slave'} at 127.19.94.193:51793
I20240502 14:13:55.941725 23098 raft_consensus.cc:482] T 80edabe29f114dd2b3b958fc954e3c2b P ff15cd84fa6640b8921bbf05dc65612e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:13:55.941931 23098 raft_consensus.cc:3037] T 80edabe29f114dd2b3b958fc954e3c2b P ff15cd84fa6640b8921bbf05dc65612e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:13:55.944537 23098 raft_consensus.cc:504] T 80edabe29f114dd2b3b958fc954e3c2b P ff15cd84fa6640b8921bbf05dc65612e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "ff15cd84fa6640b8921bbf05dc65612e" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 39713 } }
I20240502 14:13:55.946350 23098 leader_election.cc:288] T 80edabe29f114dd2b3b958fc954e3c2b P ff15cd84fa6640b8921bbf05dc65612e [CANDIDATE]: Term 2 election: Requested vote from peers 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193:38827)
I20240502 14:13:55.946774 21864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80edabe29f114dd2b3b958fc954e3c2b" candidate_uuid: "ff15cd84fa6640b8921bbf05dc65612e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0c1d247517e246c2828007ca0a6d5527"
I20240502 14:13:55.947039 21864 raft_consensus.cc:3032] T 80edabe29f114dd2b3b958fc954e3c2b P 0c1d247517e246c2828007ca0a6d5527 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:13:55.947136 21864 raft_consensus.cc:729] T 80edabe29f114dd2b3b958fc954e3c2b P 0c1d247517e246c2828007ca0a6d5527 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0c1d247517e246c2828007ca0a6d5527, State: Running, Role: LEADER
I20240502 14:13:55.947443 21864 consensus_queue.cc:260] T 80edabe29f114dd2b3b958fc954e3c2b P 0c1d247517e246c2828007ca0a6d5527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "ff15cd84fa6640b8921bbf05dc65612e" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 39713 } }
I20240502 14:13:55.947805 21864 raft_consensus.cc:3037] T 80edabe29f114dd2b3b958fc954e3c2b P 0c1d247517e246c2828007ca0a6d5527 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:13:55.950897 21864 raft_consensus.cc:2454] T 80edabe29f114dd2b3b958fc954e3c2b P 0c1d247517e246c2828007ca0a6d5527 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff15cd84fa6640b8921bbf05dc65612e in term 2.
I20240502 14:13:55.951709 23032 leader_election.cc:302] T 80edabe29f114dd2b3b958fc954e3c2b P ff15cd84fa6640b8921bbf05dc65612e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0c1d247517e246c2828007ca0a6d5527, ff15cd84fa6640b8921bbf05dc65612e; no voters:
I20240502 14:13:55.952756 24528 raft_consensus.cc:2781] T 80edabe29f114dd2b3b958fc954e3c2b P ff15cd84fa6640b8921bbf05dc65612e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:13:55.953377 24528 raft_consensus.cc:686] T 80edabe29f114dd2b3b958fc954e3c2b P ff15cd84fa6640b8921bbf05dc65612e [term 2 LEADER]: Becoming Leader. State: Replica: ff15cd84fa6640b8921bbf05dc65612e, State: Running, Role: LEADER
I20240502 14:13:55.954216 24528 consensus_queue.cc:237] T 80edabe29f114dd2b3b958fc954e3c2b P ff15cd84fa6640b8921bbf05dc65612e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "ff15cd84fa6640b8921bbf05dc65612e" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 39713 } }
I20240502 14:13:55.959439 21715 catalog_manager.cc:5484] T 80edabe29f114dd2b3b958fc954e3c2b P ff15cd84fa6640b8921bbf05dc65612e reported cstate change: term changed from 1 to 2, leader changed from 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193) to ff15cd84fa6640b8921bbf05dc65612e (127.19.94.195). New cstate: current_term: 2 leader_uuid: "ff15cd84fa6640b8921bbf05dc65612e" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "ff15cd84fa6640b8921bbf05dc65612e" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 39713 } } }
I20240502 14:13:56.031754 21844 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:13:56.044359 23611 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:56.067533 21974 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:13:56.089203 23341 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:56.098196 23078 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:56.101190 23481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:56.104949 23210 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:56.297261 21844 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:13:56.323437 23611 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:56.339021 21974 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:13:56.341159 23341 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:56.357410 23481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:56.360569 23078 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:56.371227 23210 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:56.451588 21864 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f3119216db7d4c0ea2216744e6f21c5f"
dest_uuid: "0c1d247517e246c2828007ca0a6d5527"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55256
I20240502 14:13:56.451831 21864 raft_consensus.cc:595] T f3119216db7d4c0ea2216744e6f21c5f P 0c1d247517e246c2828007ca0a6d5527 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:13:56.488616 21864 raft_consensus.cc:1261] T 80edabe29f114dd2b3b958fc954e3c2b P 0c1d247517e246c2828007ca0a6d5527 [term 2 FOLLOWER]: Refusing update from remote peer ff15cd84fa6640b8921bbf05dc65612e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:13:56.489459 24609 consensus_queue.cc:1035] T 80edabe29f114dd2b3b958fc954e3c2b P ff15cd84fa6640b8921bbf05dc65612e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:13:56.558359 21974 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:13:56.570930 21844 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:13:56.572875 23341 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:56.581250 23611 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:56.586551 23481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:56.605535 23210 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:56.624058 23078 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:56.730919 24650 raft_consensus.cc:979] T f3119216db7d4c0ea2216744e6f21c5f P 0c1d247517e246c2828007ca0a6d5527: : Instructing follower 92646999d5c64d1aae6f165593b7d14d to start an election
I20240502 14:13:56.731166 24527 raft_consensus.cc:1067] T f3119216db7d4c0ea2216744e6f21c5f P 0c1d247517e246c2828007ca0a6d5527 [term 1 LEADER]: Signalling peer 92646999d5c64d1aae6f165593b7d14d to start an election
I20240502 14:13:56.732100 23361 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f3119216db7d4c0ea2216744e6f21c5f"
dest_uuid: "92646999d5c64d1aae6f165593b7d14d"
from {username='slave'} at 127.19.94.193:49107
I20240502 14:13:56.732494 23361 raft_consensus.cc:482] T f3119216db7d4c0ea2216744e6f21c5f P 92646999d5c64d1aae6f165593b7d14d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:13:56.732641 23361 raft_consensus.cc:3037] T f3119216db7d4c0ea2216744e6f21c5f P 92646999d5c64d1aae6f165593b7d14d [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:13:56.735905 23361 raft_consensus.cc:504] T f3119216db7d4c0ea2216744e6f21c5f P 92646999d5c64d1aae6f165593b7d14d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "92646999d5c64d1aae6f165593b7d14d" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 33817 } }
I20240502 14:13:56.737541 23361 leader_election.cc:288] T f3119216db7d4c0ea2216744e6f21c5f P 92646999d5c64d1aae6f165593b7d14d [CANDIDATE]: Term 2 election: Requested vote from peers 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193:38827)
I20240502 14:13:56.738409 21864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3119216db7d4c0ea2216744e6f21c5f" candidate_uuid: "92646999d5c64d1aae6f165593b7d14d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0c1d247517e246c2828007ca0a6d5527"
I20240502 14:13:56.738624 21864 raft_consensus.cc:3032] T f3119216db7d4c0ea2216744e6f21c5f P 0c1d247517e246c2828007ca0a6d5527 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:13:56.738754 21864 raft_consensus.cc:729] T f3119216db7d4c0ea2216744e6f21c5f P 0c1d247517e246c2828007ca0a6d5527 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0c1d247517e246c2828007ca0a6d5527, State: Running, Role: LEADER
I20240502 14:13:56.739295 21864 consensus_queue.cc:260] T f3119216db7d4c0ea2216744e6f21c5f P 0c1d247517e246c2828007ca0a6d5527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "92646999d5c64d1aae6f165593b7d14d" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 33817 } }
I20240502 14:13:56.739655 21864 raft_consensus.cc:3037] T f3119216db7d4c0ea2216744e6f21c5f P 0c1d247517e246c2828007ca0a6d5527 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:13:56.742347 21864 raft_consensus.cc:2454] T f3119216db7d4c0ea2216744e6f21c5f P 0c1d247517e246c2828007ca0a6d5527 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92646999d5c64d1aae6f165593b7d14d in term 2.
I20240502 14:13:56.742991 23294 leader_election.cc:302] T f3119216db7d4c0ea2216744e6f21c5f P 92646999d5c64d1aae6f165593b7d14d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0c1d247517e246c2828007ca0a6d5527, 92646999d5c64d1aae6f165593b7d14d; no voters:
I20240502 14:13:56.743880 24651 raft_consensus.cc:2781] T f3119216db7d4c0ea2216744e6f21c5f P 92646999d5c64d1aae6f165593b7d14d [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:13:56.744560 24651 raft_consensus.cc:686] T f3119216db7d4c0ea2216744e6f21c5f P 92646999d5c64d1aae6f165593b7d14d [term 2 LEADER]: Becoming Leader. State: Replica: 92646999d5c64d1aae6f165593b7d14d, State: Running, Role: LEADER
I20240502 14:13:56.745007 24651 consensus_queue.cc:237] T f3119216db7d4c0ea2216744e6f21c5f P 92646999d5c64d1aae6f165593b7d14d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "92646999d5c64d1aae6f165593b7d14d" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 33817 } }
I20240502 14:13:56.749086 21718 catalog_manager.cc:5484] T f3119216db7d4c0ea2216744e6f21c5f P 92646999d5c64d1aae6f165593b7d14d reported cstate change: term changed from 1 to 2, leader changed from 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193) to 92646999d5c64d1aae6f165593b7d14d (127.19.94.197). New cstate: current_term: 2 leader_uuid: "92646999d5c64d1aae6f165593b7d14d" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "92646999d5c64d1aae6f165593b7d14d" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 33817 } } }
I20240502 14:13:56.819195 21844 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:13:56.829828 23341 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:56.830685 23611 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:56.839161 23210 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:56.842809 21974 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:13:56.848249 23481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:56.848707 23078 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:56.957973 21864 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8af0e06b069e424a9ace44e06f12a6d3"
dest_uuid: "0c1d247517e246c2828007ca0a6d5527"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55282
I20240502 14:13:56.958221 21864 raft_consensus.cc:595] T 8af0e06b069e424a9ace44e06f12a6d3 P 0c1d247517e246c2828007ca0a6d5527 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:13:57.077528 21844 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:13:57.078670 23611 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:57.089608 23481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:57.098392 21974 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:13:57.113034 23341 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:57.120414 23210 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:57.137538 23078 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:57.146384 21864 raft_consensus.cc:1261] T f3119216db7d4c0ea2216744e6f21c5f P 0c1d247517e246c2828007ca0a6d5527 [term 2 FOLLOWER]: Refusing update from remote peer 92646999d5c64d1aae6f165593b7d14d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:13:57.147962 24651 consensus_queue.cc:1035] T f3119216db7d4c0ea2216744e6f21c5f P 92646999d5c64d1aae6f165593b7d14d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:13:57.334440 23611 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:57.339371 21844 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:13:57.341038 21974 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:13:57.351362 23341 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:57.352700 23078 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:57.371743 23210 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:57.373843 23481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:57.440783 24761 raft_consensus.cc:979] T 8af0e06b069e424a9ace44e06f12a6d3 P 0c1d247517e246c2828007ca0a6d5527: : Instructing follower ff15cd84fa6640b8921bbf05dc65612e to start an election
I20240502 14:13:57.440996 24761 raft_consensus.cc:1067] T 8af0e06b069e424a9ace44e06f12a6d3 P 0c1d247517e246c2828007ca0a6d5527 [term 1 LEADER]: Signalling peer ff15cd84fa6640b8921bbf05dc65612e to start an election
I20240502 14:13:57.441915 23098 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8af0e06b069e424a9ace44e06f12a6d3"
dest_uuid: "ff15cd84fa6640b8921bbf05dc65612e"
from {username='slave'} at 127.19.94.193:51793
I20240502 14:13:57.442193 23098 raft_consensus.cc:482] T 8af0e06b069e424a9ace44e06f12a6d3 P ff15cd84fa6640b8921bbf05dc65612e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:13:57.442323 23098 raft_consensus.cc:3037] T 8af0e06b069e424a9ace44e06f12a6d3 P ff15cd84fa6640b8921bbf05dc65612e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:13:57.445813 23098 raft_consensus.cc:504] T 8af0e06b069e424a9ace44e06f12a6d3 P ff15cd84fa6640b8921bbf05dc65612e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "ff15cd84fa6640b8921bbf05dc65612e" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 39713 } }
I20240502 14:13:57.446899 23098 leader_election.cc:288] T 8af0e06b069e424a9ace44e06f12a6d3 P ff15cd84fa6640b8921bbf05dc65612e [CANDIDATE]: Term 2 election: Requested vote from peers 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193:38827)
I20240502 14:13:57.447173 21863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8af0e06b069e424a9ace44e06f12a6d3" candidate_uuid: "ff15cd84fa6640b8921bbf05dc65612e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0c1d247517e246c2828007ca0a6d5527"
I20240502 14:13:57.447583 21863 raft_consensus.cc:3032] T 8af0e06b069e424a9ace44e06f12a6d3 P 0c1d247517e246c2828007ca0a6d5527 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:13:57.447770 21863 raft_consensus.cc:729] T 8af0e06b069e424a9ace44e06f12a6d3 P 0c1d247517e246c2828007ca0a6d5527 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0c1d247517e246c2828007ca0a6d5527, State: Running, Role: LEADER
I20240502 14:13:57.448171 21863 consensus_queue.cc:260] T 8af0e06b069e424a9ace44e06f12a6d3 P 0c1d247517e246c2828007ca0a6d5527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "ff15cd84fa6640b8921bbf05dc65612e" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 39713 } }
I20240502 14:13:57.448626 21863 raft_consensus.cc:3037] T 8af0e06b069e424a9ace44e06f12a6d3 P 0c1d247517e246c2828007ca0a6d5527 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:13:57.452867 21863 raft_consensus.cc:2454] T 8af0e06b069e424a9ace44e06f12a6d3 P 0c1d247517e246c2828007ca0a6d5527 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff15cd84fa6640b8921bbf05dc65612e in term 2.
I20240502 14:13:57.453545 23032 leader_election.cc:302] T 8af0e06b069e424a9ace44e06f12a6d3 P ff15cd84fa6640b8921bbf05dc65612e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0c1d247517e246c2828007ca0a6d5527, ff15cd84fa6640b8921bbf05dc65612e; no voters:
I20240502 14:13:57.454002 24720 raft_consensus.cc:2781] T 8af0e06b069e424a9ace44e06f12a6d3 P ff15cd84fa6640b8921bbf05dc65612e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:13:57.454777 24720 raft_consensus.cc:686] T 8af0e06b069e424a9ace44e06f12a6d3 P ff15cd84fa6640b8921bbf05dc65612e [term 2 LEADER]: Becoming Leader. State: Replica: ff15cd84fa6640b8921bbf05dc65612e, State: Running, Role: LEADER
I20240502 14:13:57.455137 24720 consensus_queue.cc:237] T 8af0e06b069e424a9ace44e06f12a6d3 P ff15cd84fa6640b8921bbf05dc65612e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "ff15cd84fa6640b8921bbf05dc65612e" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 39713 } }
I20240502 14:13:57.458230 21715 catalog_manager.cc:5484] T 8af0e06b069e424a9ace44e06f12a6d3 P ff15cd84fa6640b8921bbf05dc65612e reported cstate change: term changed from 1 to 2, leader changed from 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193) to ff15cd84fa6640b8921bbf05dc65612e (127.19.94.195). New cstate: current_term: 2 leader_uuid: "ff15cd84fa6640b8921bbf05dc65612e" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "ff15cd84fa6640b8921bbf05dc65612e" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 39713 } } }
I20240502 14:13:57.481379 21995 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "906a106c412944e4a049f1491999ad37"
dest_uuid: "61c05b91821b450f9853ce940d18372b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58706
I20240502 14:13:57.481686 21995 raft_consensus.cc:595] T 906a106c412944e4a049f1491999ad37 P 61c05b91821b450f9853ce940d18372b [term 1 LEADER]: Received request to transfer leadership
I20240502 14:13:57.596820 21974 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:13:57.597874 21844 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:13:57.598685 23611 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:57.632705 23341 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:57.632961 23210 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:57.644739 23481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:57.669247 23078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:57.870386 24762 raft_consensus.cc:979] T 906a106c412944e4a049f1491999ad37 P 61c05b91821b450f9853ce940d18372b: : Instructing follower 5ab00105ba8d4412bfb9d009c4e40352 to start an election
I20240502 14:13:57.870576 24762 raft_consensus.cc:1067] T 906a106c412944e4a049f1491999ad37 P 61c05b91821b450f9853ce940d18372b [term 1 LEADER]: Signalling peer 5ab00105ba8d4412bfb9d009c4e40352 to start an election
I20240502 14:13:57.871304 23631 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "906a106c412944e4a049f1491999ad37"
dest_uuid: "5ab00105ba8d4412bfb9d009c4e40352"
from {username='slave'} at 127.19.94.194:44441
I20240502 14:13:57.871542 23631 raft_consensus.cc:482] T 906a106c412944e4a049f1491999ad37 P 5ab00105ba8d4412bfb9d009c4e40352 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:13:57.871655 23631 raft_consensus.cc:3037] T 906a106c412944e4a049f1491999ad37 P 5ab00105ba8d4412bfb9d009c4e40352 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:13:57.873620 23631 raft_consensus.cc:504] T 906a106c412944e4a049f1491999ad37 P 5ab00105ba8d4412bfb9d009c4e40352 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "5ab00105ba8d4412bfb9d009c4e40352" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 44599 } }
I20240502 14:13:57.874223 23631 leader_election.cc:288] T 906a106c412944e4a049f1491999ad37 P 5ab00105ba8d4412bfb9d009c4e40352 [CANDIDATE]: Term 2 election: Requested vote from peers 61c05b91821b450f9853ce940d18372b (127.19.94.194:40163)
I20240502 14:13:57.874907 21995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "906a106c412944e4a049f1491999ad37" candidate_uuid: "5ab00105ba8d4412bfb9d009c4e40352" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "61c05b91821b450f9853ce940d18372b"
I20240502 14:13:57.875116 21995 raft_consensus.cc:3032] T 906a106c412944e4a049f1491999ad37 P 61c05b91821b450f9853ce940d18372b [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:13:57.875188 21995 raft_consensus.cc:729] T 906a106c412944e4a049f1491999ad37 P 61c05b91821b450f9853ce940d18372b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 61c05b91821b450f9853ce940d18372b, State: Running, Role: LEADER
I20240502 14:13:57.875418 21995 consensus_queue.cc:260] T 906a106c412944e4a049f1491999ad37 P 61c05b91821b450f9853ce940d18372b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "5ab00105ba8d4412bfb9d009c4e40352" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 44599 } }
I20240502 14:13:57.875684 21995 raft_consensus.cc:3037] T 906a106c412944e4a049f1491999ad37 P 61c05b91821b450f9853ce940d18372b [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:13:57.877652 21995 raft_consensus.cc:2454] T 906a106c412944e4a049f1491999ad37 P 61c05b91821b450f9853ce940d18372b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ab00105ba8d4412bfb9d009c4e40352 in term 2.
I20240502 14:13:57.878219 23566 leader_election.cc:302] T 906a106c412944e4a049f1491999ad37 P 5ab00105ba8d4412bfb9d009c4e40352 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5ab00105ba8d4412bfb9d009c4e40352, 61c05b91821b450f9853ce940d18372b; no voters:
I20240502 14:13:57.878548 24394 raft_consensus.cc:2781] T 906a106c412944e4a049f1491999ad37 P 5ab00105ba8d4412bfb9d009c4e40352 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:13:57.878926 24394 raft_consensus.cc:686] T 906a106c412944e4a049f1491999ad37 P 5ab00105ba8d4412bfb9d009c4e40352 [term 2 LEADER]: Becoming Leader. State: Replica: 5ab00105ba8d4412bfb9d009c4e40352, State: Running, Role: LEADER
I20240502 14:13:57.879259 24394 consensus_queue.cc:237] T 906a106c412944e4a049f1491999ad37 P 5ab00105ba8d4412bfb9d009c4e40352 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "5ab00105ba8d4412bfb9d009c4e40352" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 44599 } }
I20240502 14:13:57.881877 21715 catalog_manager.cc:5484] T 906a106c412944e4a049f1491999ad37 P 5ab00105ba8d4412bfb9d009c4e40352 reported cstate change: term changed from 1 to 2, leader changed from 61c05b91821b450f9853ce940d18372b (127.19.94.194) to 5ab00105ba8d4412bfb9d009c4e40352 (127.19.94.199). New cstate: current_term: 2 leader_uuid: "5ab00105ba8d4412bfb9d009c4e40352" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "5ab00105ba8d4412bfb9d009c4e40352" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 44599 } } }
I20240502 14:13:57.924069 21863 raft_consensus.cc:1261] T 8af0e06b069e424a9ace44e06f12a6d3 P 0c1d247517e246c2828007ca0a6d5527 [term 2 FOLLOWER]: Refusing update from remote peer ff15cd84fa6640b8921bbf05dc65612e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:13:57.924872 24720 consensus_queue.cc:1035] T 8af0e06b069e424a9ace44e06f12a6d3 P ff15cd84fa6640b8921bbf05dc65612e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:13:58.058581 21974 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:13:58.064503 21844 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:13:58.066519 23341 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:58.068322 23611 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:58.080410 23481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:58.087185 23210 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:58.122419 23078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:58.216681 23230 consensus_queue.cc:237] T 48dfb482c48d408aa359c0b948237fd6 P f8fbf60396bb443697ce265de0eb063e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f8fbf60396bb443697ce265de0eb063e" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 41835 } }
I20240502 14:13:58.218149 24335 raft_consensus.cc:2932] T 48dfb482c48d408aa359c0b948237fd6 P f8fbf60396bb443697ce265de0eb063e [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 61c05b91821b450f9853ce940d18372b (127.19.94.194) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f8fbf60396bb443697ce265de0eb063e" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 41835 } } }
I20240502 14:13:58.223567 21717 catalog_manager.cc:5484] T 48dfb482c48d408aa359c0b948237fd6 P f8fbf60396bb443697ce265de0eb063e reported cstate change: config changed from index 4 to 6, VOTER 61c05b91821b450f9853ce940d18372b (127.19.94.194) evicted. New cstate: current_term: 2 leader_uuid: "f8fbf60396bb443697ce265de0eb063e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f8fbf60396bb443697ce265de0eb063e" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 41835 } } }
I20240502 14:13:58.228003 21974 tablet_service.cc:1514] Processing DeleteTablet for tablet 48dfb482c48d408aa359c0b948237fd6 with delete_type TABLET_DATA_TOMBSTONED (TS 61c05b91821b450f9853ce940d18372b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46584
I20240502 14:13:58.229173 24864 tablet_replica.cc:330] T 48dfb482c48d408aa359c0b948237fd6 P 61c05b91821b450f9853ce940d18372b: stopping tablet replica
I20240502 14:13:58.229935 24864 raft_consensus.cc:2229] T 48dfb482c48d408aa359c0b948237fd6 P 61c05b91821b450f9853ce940d18372b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:13:58.230463 24864 raft_consensus.cc:2258] T 48dfb482c48d408aa359c0b948237fd6 P 61c05b91821b450f9853ce940d18372b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:13:58.232852 24864 ts_tablet_manager.cc:1875] T 48dfb482c48d408aa359c0b948237fd6 P 61c05b91821b450f9853ce940d18372b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:13:58.244695 24864 ts_tablet_manager.cc:1888] T 48dfb482c48d408aa359c0b948237fd6 P 61c05b91821b450f9853ce940d18372b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 14:13:58.245191 24864 log.cc:1196] T 48dfb482c48d408aa359c0b948237fd6 P 61c05b91821b450f9853ce940d18372b: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/48dfb482c48d408aa359c0b948237fd6
I20240502 14:13:58.247418 21709 catalog_manager.cc:4830] TS 61c05b91821b450f9853ce940d18372b (127.19.94.194:40163): tablet 48dfb482c48d408aa359c0b948237fd6 (table rebalance_test_table_3 [id=a069a57795f6421096699da7bddb7a1c]) successfully deleted
I20240502 14:13:58.268815 21995 raft_consensus.cc:1261] T 906a106c412944e4a049f1491999ad37 P 61c05b91821b450f9853ce940d18372b [term 2 FOLLOWER]: Refusing update from remote peer 5ab00105ba8d4412bfb9d009c4e40352: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:13:58.269443 24394 consensus_queue.cc:1035] T 906a106c412944e4a049f1491999ad37 P 5ab00105ba8d4412bfb9d009c4e40352 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:13:58.373131 21844 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:13:58.386119 23611 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:58.386663 21974 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:13:58.391872 23481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:58.418000 23341 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:58.445837 23210 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:58.463227 23078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:58.548869 23501 consensus_queue.cc:237] T 424a2170ba374ed1982e550e8f7ebeea P d53de47e3dbd4a37bb81b478ae4d1e0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d53de47e3dbd4a37bb81b478ae4d1e0e" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 41717 } }
I20240502 14:13:58.550426 24452 raft_consensus.cc:2932] T 424a2170ba374ed1982e550e8f7ebeea P d53de47e3dbd4a37bb81b478ae4d1e0e [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d53de47e3dbd4a37bb81b478ae4d1e0e" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 41717 } } }
I20240502 14:13:58.556188 21717 catalog_manager.cc:5484] T 424a2170ba374ed1982e550e8f7ebeea P d53de47e3dbd4a37bb81b478ae4d1e0e reported cstate change: config changed from index 4 to 6, VOTER 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193) evicted. New cstate: current_term: 2 leader_uuid: "d53de47e3dbd4a37bb81b478ae4d1e0e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d53de47e3dbd4a37bb81b478ae4d1e0e" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 41717 } } }
I20240502 14:13:58.561331 21844 tablet_service.cc:1514] Processing DeleteTablet for tablet 424a2170ba374ed1982e550e8f7ebeea with delete_type TABLET_DATA_TOMBSTONED (TS 0c1d247517e246c2828007ca0a6d5527 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44122
I20240502 14:13:58.562259 24926 tablet_replica.cc:330] T 424a2170ba374ed1982e550e8f7ebeea P 0c1d247517e246c2828007ca0a6d5527: stopping tablet replica
I20240502 14:13:58.562671 24926 raft_consensus.cc:2229] T 424a2170ba374ed1982e550e8f7ebeea P 0c1d247517e246c2828007ca0a6d5527 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:13:58.562935 24926 raft_consensus.cc:2258] T 424a2170ba374ed1982e550e8f7ebeea P 0c1d247517e246c2828007ca0a6d5527 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:13:58.564211 24926 ts_tablet_manager.cc:1875] T 424a2170ba374ed1982e550e8f7ebeea P 0c1d247517e246c2828007ca0a6d5527: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:13:58.571775 24926 ts_tablet_manager.cc:1888] T 424a2170ba374ed1982e550e8f7ebeea P 0c1d247517e246c2828007ca0a6d5527: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 14:13:58.572114 24926 log.cc:1196] T 424a2170ba374ed1982e550e8f7ebeea P 0c1d247517e246c2828007ca0a6d5527: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/424a2170ba374ed1982e550e8f7ebeea
I20240502 14:13:58.573262 21708 catalog_manager.cc:4830] TS 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193:38827): tablet 424a2170ba374ed1982e550e8f7ebeea (table rebalance_test_table_4 [id=fe06048058fb45dcb952409c2f9a3754]) successfully deleted
I20240502 14:13:58.690161 21844 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:13:58.694403 23611 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:58.697067 23341 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:58.703217 23481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:58.714342 23210 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:58.717434 21974 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:13:58.725975 23078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:58.835427 23501 consensus_queue.cc:237] T 2ffa22a372c143bc87194122a407425c P d53de47e3dbd4a37bb81b478ae4d1e0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d53de47e3dbd4a37bb81b478ae4d1e0e" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 41717 } }
I20240502 14:13:58.836818 24452 raft_consensus.cc:2932] T 2ffa22a372c143bc87194122a407425c P d53de47e3dbd4a37bb81b478ae4d1e0e [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 61c05b91821b450f9853ce940d18372b (127.19.94.194) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d53de47e3dbd4a37bb81b478ae4d1e0e" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 41717 } } }
I20240502 14:13:58.841054 21717 catalog_manager.cc:5484] T 2ffa22a372c143bc87194122a407425c P d53de47e3dbd4a37bb81b478ae4d1e0e reported cstate change: config changed from index 4 to 6, VOTER 61c05b91821b450f9853ce940d18372b (127.19.94.194) evicted. New cstate: current_term: 2 leader_uuid: "d53de47e3dbd4a37bb81b478ae4d1e0e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d53de47e3dbd4a37bb81b478ae4d1e0e" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 41717 } } }
I20240502 14:13:58.845031 21974 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ffa22a372c143bc87194122a407425c with delete_type TABLET_DATA_TOMBSTONED (TS 61c05b91821b450f9853ce940d18372b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46584
I20240502 14:13:58.845710 24978 tablet_replica.cc:330] T 2ffa22a372c143bc87194122a407425c P 61c05b91821b450f9853ce940d18372b: stopping tablet replica
I20240502 14:13:58.845925 24978 raft_consensus.cc:2229] T 2ffa22a372c143bc87194122a407425c P 61c05b91821b450f9853ce940d18372b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:13:58.846166 24978 raft_consensus.cc:2258] T 2ffa22a372c143bc87194122a407425c P 61c05b91821b450f9853ce940d18372b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:13:58.846940 24978 ts_tablet_manager.cc:1875] T 2ffa22a372c143bc87194122a407425c P 61c05b91821b450f9853ce940d18372b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:13:58.852957 24978 ts_tablet_manager.cc:1888] T 2ffa22a372c143bc87194122a407425c P 61c05b91821b450f9853ce940d18372b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 14:13:58.853097 24978 log.cc:1196] T 2ffa22a372c143bc87194122a407425c P 61c05b91821b450f9853ce940d18372b: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/2ffa22a372c143bc87194122a407425c
I20240502 14:13:58.853921 21709 catalog_manager.cc:4830] TS 61c05b91821b450f9853ce940d18372b (127.19.94.194:40163): tablet 2ffa22a372c143bc87194122a407425c (table rebalance_test_table_3 [id=a069a57795f6421096699da7bddb7a1c]) successfully deleted
I20240502 14:13:58.964514 23611 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:58.965168 21844 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:13:58.981308 21974 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:13:58.987435 23341 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:59.026587 23481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:59.034452 23210 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:59.034881 23078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:59.134353 23631 consensus_queue.cc:237] T 4cc5c4909e7343578a87c1b91ed3d756 P 5ab00105ba8d4412bfb9d009c4e40352 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5ab00105ba8d4412bfb9d009c4e40352" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 44599 } }
I20240502 14:13:59.136363 24805 raft_consensus.cc:2932] T 4cc5c4909e7343578a87c1b91ed3d756 P 5ab00105ba8d4412bfb9d009c4e40352 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5ab00105ba8d4412bfb9d009c4e40352" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 44599 } } }
I20240502 14:13:59.142349 21717 catalog_manager.cc:5484] T 4cc5c4909e7343578a87c1b91ed3d756 P 5ab00105ba8d4412bfb9d009c4e40352 reported cstate change: config changed from index 4 to 6, VOTER 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193) evicted. New cstate: current_term: 2 leader_uuid: "5ab00105ba8d4412bfb9d009c4e40352" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5ab00105ba8d4412bfb9d009c4e40352" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 44599 } } }
I20240502 14:13:59.146838 21844 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cc5c4909e7343578a87c1b91ed3d756 with delete_type TABLET_DATA_TOMBSTONED (TS 0c1d247517e246c2828007ca0a6d5527 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44122
I20240502 14:13:59.147528 25030 tablet_replica.cc:330] T 4cc5c4909e7343578a87c1b91ed3d756 P 0c1d247517e246c2828007ca0a6d5527: stopping tablet replica
I20240502 14:13:59.147841 25030 raft_consensus.cc:2229] T 4cc5c4909e7343578a87c1b91ed3d756 P 0c1d247517e246c2828007ca0a6d5527 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:13:59.148104 25030 raft_consensus.cc:2258] T 4cc5c4909e7343578a87c1b91ed3d756 P 0c1d247517e246c2828007ca0a6d5527 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:13:59.148913 25030 ts_tablet_manager.cc:1875] T 4cc5c4909e7343578a87c1b91ed3d756 P 0c1d247517e246c2828007ca0a6d5527: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:13:59.155160 25030 ts_tablet_manager.cc:1888] T 4cc5c4909e7343578a87c1b91ed3d756 P 0c1d247517e246c2828007ca0a6d5527: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 14:13:59.155385 25030 log.cc:1196] T 4cc5c4909e7343578a87c1b91ed3d756 P 0c1d247517e246c2828007ca0a6d5527: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/4cc5c4909e7343578a87c1b91ed3d756
I20240502 14:13:59.156327 21708 catalog_manager.cc:4830] TS 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193:38827): tablet 4cc5c4909e7343578a87c1b91ed3d756 (table rebalance_test_table_4 [id=fe06048058fb45dcb952409c2f9a3754]) successfully deleted
I20240502 14:13:59.281131 23611 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:59.285243 21844 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:13:59.302505 21974 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:13:59.303925 23341 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:59.304898 23481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:59.313174 23210 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:59.327003 23078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:59.436091 23230 consensus_queue.cc:237] T 148d828a22194299a04c2e93003bf379 P f8fbf60396bb443697ce265de0eb063e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f8fbf60396bb443697ce265de0eb063e" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 41835 } }
I20240502 14:13:59.437458 24335 raft_consensus.cc:2932] T 148d828a22194299a04c2e93003bf379 P f8fbf60396bb443697ce265de0eb063e [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 61c05b91821b450f9853ce940d18372b (127.19.94.194) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f8fbf60396bb443697ce265de0eb063e" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 41835 } } }
I20240502 14:13:59.442131 21716 catalog_manager.cc:5484] T 148d828a22194299a04c2e93003bf379 P f8fbf60396bb443697ce265de0eb063e reported cstate change: config changed from index 4 to 6, VOTER 61c05b91821b450f9853ce940d18372b (127.19.94.194) evicted. New cstate: current_term: 2 leader_uuid: "f8fbf60396bb443697ce265de0eb063e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f8fbf60396bb443697ce265de0eb063e" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 41835 } } }
I20240502 14:13:59.445770 21974 tablet_service.cc:1514] Processing DeleteTablet for tablet 148d828a22194299a04c2e93003bf379 with delete_type TABLET_DATA_TOMBSTONED (TS 61c05b91821b450f9853ce940d18372b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46584
I20240502 14:13:59.446650 25087 tablet_replica.cc:330] T 148d828a22194299a04c2e93003bf379 P 61c05b91821b450f9853ce940d18372b: stopping tablet replica
I20240502 14:13:59.446903 25087 raft_consensus.cc:2229] T 148d828a22194299a04c2e93003bf379 P 61c05b91821b450f9853ce940d18372b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:13:59.447170 25087 raft_consensus.cc:2258] T 148d828a22194299a04c2e93003bf379 P 61c05b91821b450f9853ce940d18372b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:13:59.447958 25087 ts_tablet_manager.cc:1875] T 148d828a22194299a04c2e93003bf379 P 61c05b91821b450f9853ce940d18372b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:13:59.453924 25087 ts_tablet_manager.cc:1888] T 148d828a22194299a04c2e93003bf379 P 61c05b91821b450f9853ce940d18372b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 14:13:59.454092 25087 log.cc:1196] T 148d828a22194299a04c2e93003bf379 P 61c05b91821b450f9853ce940d18372b: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/148d828a22194299a04c2e93003bf379
I20240502 14:13:59.455024 21709 catalog_manager.cc:4830] TS 61c05b91821b450f9853ce940d18372b (127.19.94.194:40163): tablet 148d828a22194299a04c2e93003bf379 (table rebalance_test_table_1 [id=0b95f45f4b674b3a81e205cd4f1d6156]) successfully deleted
I20240502 14:13:59.560113 21844 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:13:59.571434 21974 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:13:59.574136 23611 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:59.589632 23341 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:59.606426 23210 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:59.615042 23078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:59.616621 23481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:59.719110 23098 consensus_queue.cc:237] T 80edabe29f114dd2b3b958fc954e3c2b P ff15cd84fa6640b8921bbf05dc65612e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ff15cd84fa6640b8921bbf05dc65612e" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 39713 } }
I20240502 14:13:59.721105 24763 raft_consensus.cc:2932] T 80edabe29f114dd2b3b958fc954e3c2b P ff15cd84fa6640b8921bbf05dc65612e [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ff15cd84fa6640b8921bbf05dc65612e" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 39713 } } }
I20240502 14:13:59.727919 21716 catalog_manager.cc:5484] T 80edabe29f114dd2b3b958fc954e3c2b P ff15cd84fa6640b8921bbf05dc65612e reported cstate change: config changed from index 4 to 6, VOTER 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193) evicted. New cstate: current_term: 2 leader_uuid: "ff15cd84fa6640b8921bbf05dc65612e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ff15cd84fa6640b8921bbf05dc65612e" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 39713 } } }
I20240502 14:13:59.731957 21844 tablet_service.cc:1514] Processing DeleteTablet for tablet 80edabe29f114dd2b3b958fc954e3c2b with delete_type TABLET_DATA_TOMBSTONED (TS 0c1d247517e246c2828007ca0a6d5527 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44122
I20240502 14:13:59.732744 25143 tablet_replica.cc:330] T 80edabe29f114dd2b3b958fc954e3c2b P 0c1d247517e246c2828007ca0a6d5527: stopping tablet replica
I20240502 14:13:59.732956 25143 raft_consensus.cc:2229] T 80edabe29f114dd2b3b958fc954e3c2b P 0c1d247517e246c2828007ca0a6d5527 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:13:59.733157 25143 raft_consensus.cc:2258] T 80edabe29f114dd2b3b958fc954e3c2b P 0c1d247517e246c2828007ca0a6d5527 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:13:59.733790 25143 ts_tablet_manager.cc:1875] T 80edabe29f114dd2b3b958fc954e3c2b P 0c1d247517e246c2828007ca0a6d5527: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:13:59.739140 25143 ts_tablet_manager.cc:1888] T 80edabe29f114dd2b3b958fc954e3c2b P 0c1d247517e246c2828007ca0a6d5527: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 14:13:59.739276 25143 log.cc:1196] T 80edabe29f114dd2b3b958fc954e3c2b P 0c1d247517e246c2828007ca0a6d5527: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/80edabe29f114dd2b3b958fc954e3c2b
I20240502 14:13:59.740092 21708 catalog_manager.cc:4830] TS 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193:38827): tablet 80edabe29f114dd2b3b958fc954e3c2b (table rebalance_test_table_2 [id=1f28b28258954e02b4cf5c905ea2a48c]) successfully deleted
I20240502 14:13:59.854635 21844 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:13:59.861434 21974 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:13:59.861979 23611 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:59.877596 23341 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:59.883688 23078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:59.886204 23210 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:13:59.912248 23481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:00.047370 23361 consensus_queue.cc:237] T f3119216db7d4c0ea2216744e6f21c5f P 92646999d5c64d1aae6f165593b7d14d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "92646999d5c64d1aae6f165593b7d14d" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 33817 } }
I20240502 14:14:00.049062 25177 raft_consensus.cc:2932] T f3119216db7d4c0ea2216744e6f21c5f P 92646999d5c64d1aae6f165593b7d14d [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "92646999d5c64d1aae6f165593b7d14d" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 33817 } } }
I20240502 14:14:00.054584 21718 catalog_manager.cc:5484] T f3119216db7d4c0ea2216744e6f21c5f P 92646999d5c64d1aae6f165593b7d14d reported cstate change: config changed from index 4 to 6, VOTER 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193) evicted. New cstate: current_term: 2 leader_uuid: "92646999d5c64d1aae6f165593b7d14d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "92646999d5c64d1aae6f165593b7d14d" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 33817 } } }
I20240502 14:14:00.058274 21844 tablet_service.cc:1514] Processing DeleteTablet for tablet f3119216db7d4c0ea2216744e6f21c5f with delete_type TABLET_DATA_TOMBSTONED (TS 0c1d247517e246c2828007ca0a6d5527 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44122
I20240502 14:14:00.058763 25143 tablet_replica.cc:330] T f3119216db7d4c0ea2216744e6f21c5f P 0c1d247517e246c2828007ca0a6d5527: stopping tablet replica
I20240502 14:14:00.059016 25143 raft_consensus.cc:2229] T f3119216db7d4c0ea2216744e6f21c5f P 0c1d247517e246c2828007ca0a6d5527 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:00.059249 25143 raft_consensus.cc:2258] T f3119216db7d4c0ea2216744e6f21c5f P 0c1d247517e246c2828007ca0a6d5527 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:00.059973 25143 ts_tablet_manager.cc:1875] T f3119216db7d4c0ea2216744e6f21c5f P 0c1d247517e246c2828007ca0a6d5527: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:14:00.064558 25143 ts_tablet_manager.cc:1888] T f3119216db7d4c0ea2216744e6f21c5f P 0c1d247517e246c2828007ca0a6d5527: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 14:14:00.064729 25143 log.cc:1196] T f3119216db7d4c0ea2216744e6f21c5f P 0c1d247517e246c2828007ca0a6d5527: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/f3119216db7d4c0ea2216744e6f21c5f
I20240502 14:14:00.065516 21708 catalog_manager.cc:4830] TS 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193:38827): tablet f3119216db7d4c0ea2216744e6f21c5f (table rebalance_test_table_1 [id=0b95f45f4b674b3a81e205cd4f1d6156]) successfully deleted
I20240502 14:14:00.183413 23341 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:00.183535 21844 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:00.199026 21974 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:00.199158 23210 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:00.203495 23611 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:00.211133 23481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:00.222828 23078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:00.345105 23098 consensus_queue.cc:237] T 8af0e06b069e424a9ace44e06f12a6d3 P ff15cd84fa6640b8921bbf05dc65612e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ff15cd84fa6640b8921bbf05dc65612e" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 39713 } }
I20240502 14:14:00.346598 24763 raft_consensus.cc:2932] T 8af0e06b069e424a9ace44e06f12a6d3 P ff15cd84fa6640b8921bbf05dc65612e [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ff15cd84fa6640b8921bbf05dc65612e" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 39713 } } }
I20240502 14:14:00.352783 21716 catalog_manager.cc:5484] T 8af0e06b069e424a9ace44e06f12a6d3 P ff15cd84fa6640b8921bbf05dc65612e reported cstate change: config changed from index 4 to 6, VOTER 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193) evicted. New cstate: current_term: 2 leader_uuid: "ff15cd84fa6640b8921bbf05dc65612e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ff15cd84fa6640b8921bbf05dc65612e" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 39713 } } }
I20240502 14:14:00.356770 21844 tablet_service.cc:1514] Processing DeleteTablet for tablet 8af0e06b069e424a9ace44e06f12a6d3 with delete_type TABLET_DATA_TOMBSTONED (TS 0c1d247517e246c2828007ca0a6d5527 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44122
I20240502 14:14:00.357256 25143 tablet_replica.cc:330] T 8af0e06b069e424a9ace44e06f12a6d3 P 0c1d247517e246c2828007ca0a6d5527: stopping tablet replica
I20240502 14:14:00.357475 25143 raft_consensus.cc:2229] T 8af0e06b069e424a9ace44e06f12a6d3 P 0c1d247517e246c2828007ca0a6d5527 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:00.357762 25143 raft_consensus.cc:2258] T 8af0e06b069e424a9ace44e06f12a6d3 P 0c1d247517e246c2828007ca0a6d5527 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:00.358554 25143 ts_tablet_manager.cc:1875] T 8af0e06b069e424a9ace44e06f12a6d3 P 0c1d247517e246c2828007ca0a6d5527: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:14:00.363803 25143 ts_tablet_manager.cc:1888] T 8af0e06b069e424a9ace44e06f12a6d3 P 0c1d247517e246c2828007ca0a6d5527: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 14:14:00.363909 25143 log.cc:1196] T 8af0e06b069e424a9ace44e06f12a6d3 P 0c1d247517e246c2828007ca0a6d5527: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/8af0e06b069e424a9ace44e06f12a6d3
I20240502 14:14:00.364666 21708 catalog_manager.cc:4830] TS 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193:38827): tablet 8af0e06b069e424a9ace44e06f12a6d3 (table rebalance_test_table_0 [id=ea93e5ddb3404119899abcd427b2771e]) successfully deleted
I20240502 14:14:00.469338 21844 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:00.486799 21974 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:00.487798 23611 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:00.499006 23341 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:00.502218 23210 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:00.504323 23481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:00.546437 23078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:00.631155 23631 consensus_queue.cc:237] T 906a106c412944e4a049f1491999ad37 P 5ab00105ba8d4412bfb9d009c4e40352 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5ab00105ba8d4412bfb9d009c4e40352" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 44599 } }
I20240502 14:14:00.632987 25303 raft_consensus.cc:2932] T 906a106c412944e4a049f1491999ad37 P 5ab00105ba8d4412bfb9d009c4e40352 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 61c05b91821b450f9853ce940d18372b (127.19.94.194) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5ab00105ba8d4412bfb9d009c4e40352" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 44599 } } }
I20240502 14:14:00.639055 21718 catalog_manager.cc:5484] T 906a106c412944e4a049f1491999ad37 P 5ab00105ba8d4412bfb9d009c4e40352 reported cstate change: config changed from index 4 to 6, VOTER 61c05b91821b450f9853ce940d18372b (127.19.94.194) evicted. New cstate: current_term: 2 leader_uuid: "5ab00105ba8d4412bfb9d009c4e40352" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5ab00105ba8d4412bfb9d009c4e40352" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 44599 } } }
I20240502 14:14:00.643399 21974 tablet_service.cc:1514] Processing DeleteTablet for tablet 906a106c412944e4a049f1491999ad37 with delete_type TABLET_DATA_TOMBSTONED (TS 61c05b91821b450f9853ce940d18372b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46584
I20240502 14:14:00.644080 25305 tablet_replica.cc:330] T 906a106c412944e4a049f1491999ad37 P 61c05b91821b450f9853ce940d18372b: stopping tablet replica
I20240502 14:14:00.644241 25305 raft_consensus.cc:2229] T 906a106c412944e4a049f1491999ad37 P 61c05b91821b450f9853ce940d18372b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:00.644481 25305 raft_consensus.cc:2258] T 906a106c412944e4a049f1491999ad37 P 61c05b91821b450f9853ce940d18372b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:00.645083 25305 ts_tablet_manager.cc:1875] T 906a106c412944e4a049f1491999ad37 P 61c05b91821b450f9853ce940d18372b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:14:00.650187 25305 ts_tablet_manager.cc:1888] T 906a106c412944e4a049f1491999ad37 P 61c05b91821b450f9853ce940d18372b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 14:14:00.650293 25305 log.cc:1196] T 906a106c412944e4a049f1491999ad37 P 61c05b91821b450f9853ce940d18372b: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/906a106c412944e4a049f1491999ad37
I20240502 14:14:00.651038 21709 catalog_manager.cc:4830] TS 61c05b91821b450f9853ce940d18372b (127.19.94.194:40163): tablet 906a106c412944e4a049f1491999ad37 (table rebalance_test_table_2 [id=1f28b28258954e02b4cf5c905ea2a48c]) successfully deleted
I20240502 14:14:01.066923 23611 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:01.073202 21844 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:01.119304 23341 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:01.121727 21974 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:01.133410 23481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:01.133610 23210 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:01.135331 23078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:01.312327 21844 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:01.326907 21974 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:01.327623 23611 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:01.337584 23341 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:01.348624 23078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:01.359869 23481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:01.370764 23210 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:02.186807 21844 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:02.193316 21974 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:02.204820 23341 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:02.210165 23611 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:02.224749 23078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:02.225854 23210 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:02.237872 23481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:02.478467 21974 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:02.482064 23481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:02.486922 23611 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:02.491292 23341 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:02.498958 23210 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:02.501346 21844 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:02.517499 23078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:02.817328 21844 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:02.818492 23611 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:02.821895 21974 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:02.830993 23210 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:02.843501 23481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:02.846819 23341 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:02.873385 23078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
66ce354883f14c8392e7fc8e9383bc85 | 127.19.94.254:40967 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.19.94.212:39595 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
0c1d247517e246c2828007ca0a6d5527 | 127.19.94.193:38827 | HEALTHY | <none> | 3 | 0
5ab00105ba8d4412bfb9d009c4e40352 | 127.19.94.199:44599 | HEALTHY | <none> | 2 | 0
61c05b91821b450f9853ce940d18372b | 127.19.94.194:40163 | HEALTHY | <none> | 3 | 0
92646999d5c64d1aae6f165593b7d14d | 127.19.94.197:33817 | HEALTHY | <none> | 2 | 0
d53de47e3dbd4a37bb81b478ae4d1e0e | 127.19.94.198:41717 | HEALTHY | <none> | 2 | 0
f8fbf60396bb443697ce265de0eb063e | 127.19.94.196:41835 | HEALTHY | <none> | 2 | 0
ff15cd84fa6640b8921bbf05dc65612e | 127.19.94.195:39713 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.19.94.193 | experimental | 127.19.94.193:38827
local_ip_for_outbound_sockets | 127.19.94.194 | experimental | 127.19.94.194:40163
local_ip_for_outbound_sockets | 127.19.94.195 | experimental | 127.19.94.195:39713
local_ip_for_outbound_sockets | 127.19.94.196 | experimental | 127.19.94.196:41835
local_ip_for_outbound_sockets | 127.19.94.197 | experimental | 127.19.94.197:33817
local_ip_for_outbound_sockets | 127.19.94.198 | experimental | 127.19.94.198:41717
local_ip_for_outbound_sockets | 127.19.94.199 | experimental | 127.19.94.199:44599
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.19.94.193:38827
server_dump_info_path | /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.94.194:40163
server_dump_info_path | /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.94.195:39713
server_dump_info_path | /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.19.94.196:41835
server_dump_info_path | /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.94.197:33817
server_dump_info_path | /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.94.198:41717
server_dump_info_path | /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.19.94.199:44599
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.19.94.212:39595 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 14:14:02.939764 19835 log_verifier.cc:126] Checking tablet 148d828a22194299a04c2e93003bf379
I20240502 14:14:02.959013 19835 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 148d828a22194299a04c2e93003bf379
I20240502 14:14:02.959177 19835 log_verifier.cc:126] Checking tablet 159f343b034a47e88f5d8ecfa0bc282c
I20240502 14:14:02.971593 19835 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 159f343b034a47e88f5d8ecfa0bc282c
I20240502 14:14:02.971709 19835 log_verifier.cc:126] Checking tablet 2ffa22a372c143bc87194122a407425c
I20240502 14:14:02.984329 19835 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 2ffa22a372c143bc87194122a407425c
I20240502 14:14:02.984490 19835 log_verifier.cc:126] Checking tablet 41b085facc334a7da0c7dd937275009f
I20240502 14:14:02.992584 19835 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 41b085facc334a7da0c7dd937275009f
I20240502 14:14:02.992714 19835 log_verifier.cc:126] Checking tablet 424a2170ba374ed1982e550e8f7ebeea
I20240502 14:14:03.005285 19835 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 424a2170ba374ed1982e550e8f7ebeea
I20240502 14:14:03.005419 19835 log_verifier.cc:126] Checking tablet 48dfb482c48d408aa359c0b948237fd6
I20240502 14:14:03.018637 19835 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 48dfb482c48d408aa359c0b948237fd6
I20240502 14:14:03.018782 19835 log_verifier.cc:126] Checking tablet 4cc5c4909e7343578a87c1b91ed3d756
I20240502 14:14:03.031832 19835 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 4cc5c4909e7343578a87c1b91ed3d756
I20240502 14:14:03.031978 19835 log_verifier.cc:126] Checking tablet 54ec5887ceee418c9915a1f043978f83
I20240502 14:14:03.044019 19835 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 54ec5887ceee418c9915a1f043978f83
I20240502 14:14:03.044147 19835 log_verifier.cc:126] Checking tablet 7253bd03818a4c2f9f0478aebb326273
I20240502 14:14:03.056628 19835 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 7253bd03818a4c2f9f0478aebb326273
I20240502 14:14:03.056763 19835 log_verifier.cc:126] Checking tablet 80edabe29f114dd2b3b958fc954e3c2b
I20240502 14:14:03.070127 19835 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 80edabe29f114dd2b3b958fc954e3c2b
I20240502 14:14:03.070256 19835 log_verifier.cc:126] Checking tablet 8a2d3ba227dd43eb941b7e1b3e8445ce
I20240502 14:14:03.082093 19835 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 8a2d3ba227dd43eb941b7e1b3e8445ce
I20240502 14:14:03.082214 19835 log_verifier.cc:126] Checking tablet 8af0e06b069e424a9ace44e06f12a6d3
I20240502 14:14:03.096762 19835 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 8af0e06b069e424a9ace44e06f12a6d3
I20240502 14:14:03.096906 19835 log_verifier.cc:126] Checking tablet 906a106c412944e4a049f1491999ad37
I20240502 14:14:03.109261 19835 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 906a106c412944e4a049f1491999ad37
I20240502 14:14:03.109407 19835 log_verifier.cc:126] Checking tablet 9597a33811ef4247a4c0ecb76d0afbaf
I20240502 14:14:03.121767 19835 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 9597a33811ef4247a4c0ecb76d0afbaf
I20240502 14:14:03.121933 19835 log_verifier.cc:126] Checking tablet ab54535b58be4e518340f33449bcdb26
I20240502 14:14:03.136011 19835 log_verifier.cc:177] Verified matching terms for 3 ops in tablet ab54535b58be4e518340f33449bcdb26
I20240502 14:14:03.136147 19835 log_verifier.cc:126] Checking tablet f3119216db7d4c0ea2216744e6f21c5f
I20240502 14:14:03.148823 19835 log_verifier.cc:177] Verified matching terms for 6 ops in tablet f3119216db7d4c0ea2216744e6f21c5f
I20240502 14:14:03.152140 19835 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_zeJNl/build/asan/bin/kudu
/tmp/dist-test-task_zeJNl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.94.200:0
--local_ip_for_outbound_sockets=127.19.94.200
--webserver_interface=127.19.94.200
--webserver_port=0
--tserver_master_addrs=127.19.94.254:40967
--builtin_ntp_servers=127.19.94.212:39595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 14:14:03.539300 25533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:03.539690 25533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:14:03.539870 25533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:03.548935 25533 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:14:03.549077 25533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:14:03.549229 25533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.94.200
I20240502 14:14:03.561251 25533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.94.212:39595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.94.200:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.19.94.200
--webserver_port=0
--tserver_master_addrs=127.19.94.254:40967
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.94.200
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:14:03.562773 25533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:14:03.564666 25533 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:03.578585 25538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:03.579746 25539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:03.581327 25541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:03.585928 25533 server_base.cc:1034] running on GCE node
I20240502 14:14:03.586654 25533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:14:03.587774 25533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:14:03.589036 25533 hybrid_clock.cc:648] HybridClock initialized: now 1714659243588972 us; error 58 us; skew 500 ppm
I20240502 14:14:03.589403 25533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:03.592260 25533 webserver.cc:460] Webserver started at http://127.19.94.200:39251/ using document root <none> and password file <none>
I20240502 14:14:03.593397 25533 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:03.593533 25533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:03.593895 25533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:03.596665 25533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "9666ed56f267401c9c501ad38c2cad1c"
format_stamp: "Formatted at 2024-05-02 14:14:03 on dist-test-slave-3j2v"
I20240502 14:14:03.597457 25533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "9666ed56f267401c9c501ad38c2cad1c"
format_stamp: "Formatted at 2024-05-02 14:14:03 on dist-test-slave-3j2v"
I20240502 14:14:03.602993 25533 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 14:14:03.606909 25547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:03.608544 25533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:14:03.608736 25533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "9666ed56f267401c9c501ad38c2cad1c"
format_stamp: "Formatted at 2024-05-02 14:14:03 on dist-test-slave-3j2v"
I20240502 14:14:03.608958 25533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:03.627504 25533 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:03.628525 25533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:14:03.628831 25533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:03.629922 25533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:03.631839 25533 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:03.631964 25533 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:03.632097 25533 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:03.632174 25533 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:03.673586 25533 rpc_server.cc:307] RPC server started. Bound to: 127.19.94.200:43735
I20240502 14:14:03.673645 25659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.94.200:43735 every 8 connection(s)
I20240502 14:14:03.674902 25533 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 14:14:03.685317 19835 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_zeJNl/build/asan/bin/kudu as pid 25533
I20240502 14:14:03.685634 19835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 14:14:03.687944 25660 heartbeater.cc:347] Connected to a master server at 127.19.94.254:40967
I20240502 14:14:03.688232 25660 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:03.689035 25660 heartbeater.cc:511] Master 127.19.94.254:40967 requested a full tablet report, sending...
I20240502 14:14:03.690461 21722 ts_manager.cc:194] Registered new tserver with Master: 9666ed56f267401c9c501ad38c2cad1c (127.19.94.200:43735)
I20240502 14:14:03.691673 21722 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.94.200:45919
I20240502 14:14:04.225070 23611 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:04.236881 21844 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:04.260125 21974 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:04.262058 23341 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:04.297588 25595 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:14:04.321816 23481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:04.329521 23210 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:04.330245 23078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:04.534446 23611 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:04.541764 21844 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:04.555341 23341 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:04.556186 23481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:04.558861 21974 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:04.559810 25595 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:14:04.585194 23210 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:04.611620 23078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:04.695955 25660 heartbeater.cc:503] Master 127.19.94.254:40967 was elected leader, sending a full tablet report...
I20240502 14:14:04.726212 21995 consensus_queue.cc:237] T 9597a33811ef4247a4c0ecb76d0afbaf P 61c05b91821b450f9853ce940d18372b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "9666ed56f267401c9c501ad38c2cad1c" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 43735 } }
W20240502 14:14:04.735220 21929 consensus_peers.cc:485] T 9597a33811ef4247a4c0ecb76d0afbaf P 61c05b91821b450f9853ce940d18372b -> Peer 9666ed56f267401c9c501ad38c2cad1c (127.19.94.200:43735): Couldn't send request to peer 9666ed56f267401c9c501ad38c2cad1c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9597a33811ef4247a4c0ecb76d0afbaf. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:05.162494 25765 ts_tablet_manager.cc:902] T 9597a33811ef4247a4c0ecb76d0afbaf P 9666ed56f267401c9c501ad38c2cad1c: Initiating tablet copy from peer 61c05b91821b450f9853ce940d18372b (127.19.94.194:40163)
I20240502 14:14:05.164328 25765 tablet_copy_client.cc:308] T 9597a33811ef4247a4c0ecb76d0afbaf P 9666ed56f267401c9c501ad38c2cad1c: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.194:40163
I20240502 14:14:05.170836 22015 tablet_copy_service.cc:140] P 61c05b91821b450f9853ce940d18372b: Received BeginTabletCopySession request for tablet 9597a33811ef4247a4c0ecb76d0afbaf from peer 9666ed56f267401c9c501ad38c2cad1c ({username='slave'} at 127.19.94.200:53541)
I20240502 14:14:05.171001 22015 tablet_copy_service.cc:161] P 61c05b91821b450f9853ce940d18372b: Beginning new tablet copy session on tablet 9597a33811ef4247a4c0ecb76d0afbaf from peer 9666ed56f267401c9c501ad38c2cad1c at {username='slave'} at 127.19.94.200:53541: session id = 9666ed56f267401c9c501ad38c2cad1c-9597a33811ef4247a4c0ecb76d0afbaf
I20240502 14:14:05.173134 22015 tablet_copy_source_session.cc:205] T 9597a33811ef4247a4c0ecb76d0afbaf P 61c05b91821b450f9853ce940d18372b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:05.177533 25765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9597a33811ef4247a4c0ecb76d0afbaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:05.187242 25765 tablet_copy_client.cc:783] T 9597a33811ef4247a4c0ecb76d0afbaf P 9666ed56f267401c9c501ad38c2cad1c: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:05.187604 25765 tablet_copy_client.cc:647] T 9597a33811ef4247a4c0ecb76d0afbaf P 9666ed56f267401c9c501ad38c2cad1c: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:05.189853 25765 tablet_copy_client.cc:523] T 9597a33811ef4247a4c0ecb76d0afbaf P 9666ed56f267401c9c501ad38c2cad1c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:05.193305 25765 tablet_bootstrap.cc:492] T 9597a33811ef4247a4c0ecb76d0afbaf P 9666ed56f267401c9c501ad38c2cad1c: Bootstrap starting.
I20240502 14:14:05.203619 25765 log.cc:824] T 9597a33811ef4247a4c0ecb76d0afbaf P 9666ed56f267401c9c501ad38c2cad1c: Log is configured to *not* fsync() on all Append() calls
I20240502 14:14:05.236420 25765 tablet_bootstrap.cc:492] T 9597a33811ef4247a4c0ecb76d0afbaf P 9666ed56f267401c9c501ad38c2cad1c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:14:05.237072 25765 tablet_bootstrap.cc:492] T 9597a33811ef4247a4c0ecb76d0afbaf P 9666ed56f267401c9c501ad38c2cad1c: Bootstrap complete.
I20240502 14:14:05.237543 25765 ts_tablet_manager.cc:1367] T 9597a33811ef4247a4c0ecb76d0afbaf P 9666ed56f267401c9c501ad38c2cad1c: Time spent bootstrapping tablet: real 0.044s user 0.035s sys 0.008s
I20240502 14:14:05.242432 25765 raft_consensus.cc:348] T 9597a33811ef4247a4c0ecb76d0afbaf P 9666ed56f267401c9c501ad38c2cad1c [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } }
I20240502 14:14:05.243363 25765 raft_consensus.cc:729] T 9597a33811ef4247a4c0ecb76d0afbaf P 9666ed56f267401c9c501ad38c2cad1c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9666ed56f267401c9c501ad38c2cad1c, State: Initialized, Role: FOLLOWER
I20240502 14:14:05.243809 25765 consensus_queue.cc:260] T 9597a33811ef4247a4c0ecb76d0afbaf P 9666ed56f267401c9c501ad38c2cad1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "9666ed56f267401c9c501ad38c2cad1c" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 43735 } }
I20240502 14:14:05.245600 25765 ts_tablet_manager.cc:1398] T 9597a33811ef4247a4c0ecb76d0afbaf P 9666ed56f267401c9c501ad38c2cad1c: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20240502 14:14:05.246628 22015 tablet_copy_service.cc:342] P 61c05b91821b450f9853ce940d18372b: Request end of tablet copy session 9666ed56f267401c9c501ad38c2cad1c-9597a33811ef4247a4c0ecb76d0afbaf received from {username='slave'} at 127.19.94.200:53541
I20240502 14:14:05.246774 22015 tablet_copy_service.cc:433] P 61c05b91821b450f9853ce940d18372b: ending tablet copy session 9666ed56f267401c9c501ad38c2cad1c-9597a33811ef4247a4c0ecb76d0afbaf on tablet 9597a33811ef4247a4c0ecb76d0afbaf with peer 9666ed56f267401c9c501ad38c2cad1c
I20240502 14:14:05.723414 25615 raft_consensus.cc:1203] T 9597a33811ef4247a4c0ecb76d0afbaf P 9666ed56f267401c9c501ad38c2cad1c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:05.725312 25774 raft_consensus.cc:2932] T 9597a33811ef4247a4c0ecb76d0afbaf P 61c05b91821b450f9853ce940d18372b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9666ed56f267401c9c501ad38c2cad1c (127.19.94.200) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "9666ed56f267401c9c501ad38c2cad1c" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 43735 } } }
I20240502 14:14:05.726269 25615 raft_consensus.cc:2932] T 9597a33811ef4247a4c0ecb76d0afbaf P 9666ed56f267401c9c501ad38c2cad1c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9666ed56f267401c9c501ad38c2cad1c (127.19.94.200) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "9666ed56f267401c9c501ad38c2cad1c" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 43735 } } }
I20240502 14:14:05.729076 25774 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:39410 (request call id 0) took 1003 ms. Trace:
I20240502 14:14:05.729203 25774 rpcz_store.cc:276] 0502 14:14:04.725612 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 14:14:04.725690 (+ 78us) service_pool.cc:225] Handling call
0502 14:14:04.726897 (+ 1207us) log.cc:842] Serialized 260 byte log entry
0502 14:14:05.729000 (+1002103us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":83,"raft.queue_time_us":396,"raft.run_cpu_time_us":322,"raft.run_wall_time_us":323,"spinlock_wait_cycles":120192,"thread_start_us":562,"threads_started":2,"wal-append.queue_time_us":349,"wal-append.run_cpu_time_us":517,"wal-append.run_wall_time_us":1000627}
I20240502 14:14:05.731490 21717 catalog_manager.cc:5484] T 9597a33811ef4247a4c0ecb76d0afbaf P 61c05b91821b450f9853ce940d18372b reported cstate change: config changed from index -1 to 4, VOTER 9666ed56f267401c9c501ad38c2cad1c (127.19.94.200) added. New cstate: current_term: 1 leader_uuid: "61c05b91821b450f9853ce940d18372b" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "9666ed56f267401c9c501ad38c2cad1c" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 43735 } } }
I20240502 14:14:05.763806 21863 consensus_queue.cc:237] T 7253bd03818a4c2f9f0478aebb326273 P 0c1d247517e246c2828007ca0a6d5527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "9666ed56f267401c9c501ad38c2cad1c" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 43735 } }
W20240502 14:14:05.772025 21798 consensus_peers.cc:485] T 7253bd03818a4c2f9f0478aebb326273 P 0c1d247517e246c2828007ca0a6d5527 -> Peer 9666ed56f267401c9c501ad38c2cad1c (127.19.94.200:43735): Couldn't send request to peer 9666ed56f267401c9c501ad38c2cad1c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7253bd03818a4c2f9f0478aebb326273. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:06.305716 25803 ts_tablet_manager.cc:902] T 7253bd03818a4c2f9f0478aebb326273 P 9666ed56f267401c9c501ad38c2cad1c: Initiating tablet copy from peer 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193:38827)
I20240502 14:14:06.306897 25803 tablet_copy_client.cc:308] T 7253bd03818a4c2f9f0478aebb326273 P 9666ed56f267401c9c501ad38c2cad1c: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.193:38827
I20240502 14:14:06.313155 21884 tablet_copy_service.cc:140] P 0c1d247517e246c2828007ca0a6d5527: Received BeginTabletCopySession request for tablet 7253bd03818a4c2f9f0478aebb326273 from peer 9666ed56f267401c9c501ad38c2cad1c ({username='slave'} at 127.19.94.200:33243)
I20240502 14:14:06.313362 21884 tablet_copy_service.cc:161] P 0c1d247517e246c2828007ca0a6d5527: Beginning new tablet copy session on tablet 7253bd03818a4c2f9f0478aebb326273 from peer 9666ed56f267401c9c501ad38c2cad1c at {username='slave'} at 127.19.94.200:33243: session id = 9666ed56f267401c9c501ad38c2cad1c-7253bd03818a4c2f9f0478aebb326273
I20240502 14:14:06.315518 21884 tablet_copy_source_session.cc:205] T 7253bd03818a4c2f9f0478aebb326273 P 0c1d247517e246c2828007ca0a6d5527: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:06.316867 25803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7253bd03818a4c2f9f0478aebb326273. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:06.321367 25803 tablet_copy_client.cc:783] T 7253bd03818a4c2f9f0478aebb326273 P 9666ed56f267401c9c501ad38c2cad1c: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:06.321619 25803 tablet_copy_client.cc:647] T 7253bd03818a4c2f9f0478aebb326273 P 9666ed56f267401c9c501ad38c2cad1c: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:06.323451 25803 tablet_copy_client.cc:523] T 7253bd03818a4c2f9f0478aebb326273 P 9666ed56f267401c9c501ad38c2cad1c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:06.326018 25803 tablet_bootstrap.cc:492] T 7253bd03818a4c2f9f0478aebb326273 P 9666ed56f267401c9c501ad38c2cad1c: Bootstrap starting.
I20240502 14:14:06.357952 25803 tablet_bootstrap.cc:492] T 7253bd03818a4c2f9f0478aebb326273 P 9666ed56f267401c9c501ad38c2cad1c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:14:06.358415 25803 tablet_bootstrap.cc:492] T 7253bd03818a4c2f9f0478aebb326273 P 9666ed56f267401c9c501ad38c2cad1c: Bootstrap complete.
I20240502 14:14:06.358690 25803 ts_tablet_manager.cc:1367] T 7253bd03818a4c2f9f0478aebb326273 P 9666ed56f267401c9c501ad38c2cad1c: Time spent bootstrapping tablet: real 0.033s user 0.024s sys 0.008s
I20240502 14:14:06.359613 25803 raft_consensus.cc:348] T 7253bd03818a4c2f9f0478aebb326273 P 9666ed56f267401c9c501ad38c2cad1c [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } }
I20240502 14:14:06.360054 25803 raft_consensus.cc:729] T 7253bd03818a4c2f9f0478aebb326273 P 9666ed56f267401c9c501ad38c2cad1c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9666ed56f267401c9c501ad38c2cad1c, State: Initialized, Role: FOLLOWER
I20240502 14:14:06.360248 25803 consensus_queue.cc:260] T 7253bd03818a4c2f9f0478aebb326273 P 9666ed56f267401c9c501ad38c2cad1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "9666ed56f267401c9c501ad38c2cad1c" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 43735 } }
I20240502 14:14:06.361095 25803 ts_tablet_manager.cc:1398] T 7253bd03818a4c2f9f0478aebb326273 P 9666ed56f267401c9c501ad38c2cad1c: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 14:14:06.362048 21884 tablet_copy_service.cc:342] P 0c1d247517e246c2828007ca0a6d5527: Request end of tablet copy session 9666ed56f267401c9c501ad38c2cad1c-7253bd03818a4c2f9f0478aebb326273 received from {username='slave'} at 127.19.94.200:33243
I20240502 14:14:06.362164 21884 tablet_copy_service.cc:433] P 0c1d247517e246c2828007ca0a6d5527: ending tablet copy session 9666ed56f267401c9c501ad38c2cad1c-7253bd03818a4c2f9f0478aebb326273 on tablet 7253bd03818a4c2f9f0478aebb326273 with peer 9666ed56f267401c9c501ad38c2cad1c
I20240502 14:14:06.914392 25615 raft_consensus.cc:1203] T 7253bd03818a4c2f9f0478aebb326273 P 9666ed56f267401c9c501ad38c2cad1c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:06.916463 25815 raft_consensus.cc:2932] T 7253bd03818a4c2f9f0478aebb326273 P 0c1d247517e246c2828007ca0a6d5527 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9666ed56f267401c9c501ad38c2cad1c (127.19.94.200) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "9666ed56f267401c9c501ad38c2cad1c" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 43735 } } }
I20240502 14:14:06.917340 25615 raft_consensus.cc:2932] T 7253bd03818a4c2f9f0478aebb326273 P 9666ed56f267401c9c501ad38c2cad1c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9666ed56f267401c9c501ad38c2cad1c (127.19.94.200) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "9666ed56f267401c9c501ad38c2cad1c" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 43735 } } }
I20240502 14:14:06.920892 25815 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:53096 (request call id 0) took 1157 ms. Trace:
I20240502 14:14:06.921128 25815 rpcz_store.cc:276] 0502 14:14:05.763184 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 14:14:05.763280 (+ 96us) service_pool.cc:225] Handling call
0502 14:14:05.764590 (+ 1310us) log.cc:842] Serialized 260 byte log entry
0502 14:14:06.920833 (+1156243us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":73,"raft.queue_time_us":381,"raft.run_cpu_time_us":277,"raft.run_wall_time_us":277,"thread_start_us":531,"threads_started":2,"wal-append.queue_time_us":375,"wal-append.run_cpu_time_us":614,"wal-append.run_wall_time_us":1000675}
I20240502 14:14:06.923678 21715 catalog_manager.cc:5484] T 7253bd03818a4c2f9f0478aebb326273 P 9666ed56f267401c9c501ad38c2cad1c reported cstate change: config changed from index -1 to 4, VOTER 9666ed56f267401c9c501ad38c2cad1c (127.19.94.200) added. New cstate: current_term: 1 leader_uuid: "0c1d247517e246c2828007ca0a6d5527" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "9666ed56f267401c9c501ad38c2cad1c" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 43735 } } }
I20240502 14:14:07.054311 23611 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:07.058061 21844 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:07.071471 25595 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:14:07.077937 23341 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:07.081131 21974 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:07.086318 23210 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:07.096864 23481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:07.118872 23078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:07.199205 21863 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7253bd03818a4c2f9f0478aebb326273"
dest_uuid: "0c1d247517e246c2828007ca0a6d5527"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53128
I20240502 14:14:07.199424 21863 raft_consensus.cc:595] T 7253bd03818a4c2f9f0478aebb326273 P 0c1d247517e246c2828007ca0a6d5527 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:14:07.296916 21844 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:07.301384 23611 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:07.323168 25595 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:14:07.333159 21974 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:07.379336 23210 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:07.379515 23341 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:07.379905 23481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:07.379948 23078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:07.406096 25813 raft_consensus.cc:979] T 7253bd03818a4c2f9f0478aebb326273 P 0c1d247517e246c2828007ca0a6d5527: : Instructing follower 9666ed56f267401c9c501ad38c2cad1c to start an election
I20240502 14:14:07.406270 25813 raft_consensus.cc:1067] T 7253bd03818a4c2f9f0478aebb326273 P 0c1d247517e246c2828007ca0a6d5527 [term 1 LEADER]: Signalling peer 9666ed56f267401c9c501ad38c2cad1c to start an election
I20240502 14:14:07.407181 25615 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7253bd03818a4c2f9f0478aebb326273"
dest_uuid: "9666ed56f267401c9c501ad38c2cad1c"
from {username='slave'} at 127.19.94.193:59451
I20240502 14:14:07.407495 25615 raft_consensus.cc:482] T 7253bd03818a4c2f9f0478aebb326273 P 9666ed56f267401c9c501ad38c2cad1c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:14:07.407675 25615 raft_consensus.cc:3037] T 7253bd03818a4c2f9f0478aebb326273 P 9666ed56f267401c9c501ad38c2cad1c [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:14:07.410676 25615 raft_consensus.cc:504] T 7253bd03818a4c2f9f0478aebb326273 P 9666ed56f267401c9c501ad38c2cad1c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "9666ed56f267401c9c501ad38c2cad1c" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 43735 } }
I20240502 14:14:07.413645 21863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7253bd03818a4c2f9f0478aebb326273" candidate_uuid: "9666ed56f267401c9c501ad38c2cad1c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0c1d247517e246c2828007ca0a6d5527"
I20240502 14:14:07.413870 21863 raft_consensus.cc:3032] T 7253bd03818a4c2f9f0478aebb326273 P 0c1d247517e246c2828007ca0a6d5527 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:14:07.413960 21863 raft_consensus.cc:729] T 7253bd03818a4c2f9f0478aebb326273 P 0c1d247517e246c2828007ca0a6d5527 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0c1d247517e246c2828007ca0a6d5527, State: Running, Role: LEADER
I20240502 14:14:07.414245 21863 consensus_queue.cc:260] T 7253bd03818a4c2f9f0478aebb326273 P 0c1d247517e246c2828007ca0a6d5527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "9666ed56f267401c9c501ad38c2cad1c" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 43735 } }
I20240502 14:14:07.414481 21863 raft_consensus.cc:3037] T 7253bd03818a4c2f9f0478aebb326273 P 0c1d247517e246c2828007ca0a6d5527 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:14:07.418190 21863 raft_consensus.cc:2454] T 7253bd03818a4c2f9f0478aebb326273 P 0c1d247517e246c2828007ca0a6d5527 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9666ed56f267401c9c501ad38c2cad1c in term 2.
I20240502 14:14:07.418923 25549 leader_election.cc:302] T 7253bd03818a4c2f9f0478aebb326273 P 9666ed56f267401c9c501ad38c2cad1c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0c1d247517e246c2828007ca0a6d5527, 9666ed56f267401c9c501ad38c2cad1c; no voters:
I20240502 14:14:07.419399 25814 raft_consensus.cc:2781] T 7253bd03818a4c2f9f0478aebb326273 P 9666ed56f267401c9c501ad38c2cad1c [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:14:07.422380 25615 leader_election.cc:288] T 7253bd03818a4c2f9f0478aebb326273 P 9666ed56f267401c9c501ad38c2cad1c [CANDIDATE]: Term 2 election: Requested vote from peers 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193:38827)
I20240502 14:14:07.438480 25814 raft_consensus.cc:686] T 7253bd03818a4c2f9f0478aebb326273 P 9666ed56f267401c9c501ad38c2cad1c [term 2 LEADER]: Becoming Leader. State: Replica: 9666ed56f267401c9c501ad38c2cad1c, State: Running, Role: LEADER
I20240502 14:14:07.439440 25814 consensus_queue.cc:237] T 7253bd03818a4c2f9f0478aebb326273 P 9666ed56f267401c9c501ad38c2cad1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "9666ed56f267401c9c501ad38c2cad1c" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 43735 } }
I20240502 14:14:07.444290 21715 catalog_manager.cc:5484] T 7253bd03818a4c2f9f0478aebb326273 P 9666ed56f267401c9c501ad38c2cad1c reported cstate change: term changed from 1 to 2, leader changed from 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193) to 9666ed56f267401c9c501ad38c2cad1c (127.19.94.200). New cstate: current_term: 2 leader_uuid: "9666ed56f267401c9c501ad38c2cad1c" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 } } peers { permanent_uuid: "9666ed56f267401c9c501ad38c2cad1c" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 43735 } } }
I20240502 14:14:07.598632 21844 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:07.603039 23611 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:07.612942 21974 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:07.626448 23341 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:07.652204 25595 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:14:07.660130 23481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:07.674908 23210 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:07.711284 23078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:07.765899 21995 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9597a33811ef4247a4c0ecb76d0afbaf"
dest_uuid: "61c05b91821b450f9853ce940d18372b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39450
I20240502 14:14:07.766170 21995 raft_consensus.cc:595] T 9597a33811ef4247a4c0ecb76d0afbaf P 61c05b91821b450f9853ce940d18372b [term 1 LEADER]: Received request to transfer leadership
I20240502 14:14:07.805383 25970 raft_consensus.cc:979] T 9597a33811ef4247a4c0ecb76d0afbaf P 61c05b91821b450f9853ce940d18372b: : Instructing follower 9666ed56f267401c9c501ad38c2cad1c to start an election
I20240502 14:14:07.805562 25970 raft_consensus.cc:1067] T 9597a33811ef4247a4c0ecb76d0afbaf P 61c05b91821b450f9853ce940d18372b [term 1 LEADER]: Signalling peer 9666ed56f267401c9c501ad38c2cad1c to start an election
I20240502 14:14:07.806257 25615 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9597a33811ef4247a4c0ecb76d0afbaf"
dest_uuid: "9666ed56f267401c9c501ad38c2cad1c"
from {username='slave'} at 127.19.94.194:51659
I20240502 14:14:07.806514 25615 raft_consensus.cc:482] T 9597a33811ef4247a4c0ecb76d0afbaf P 9666ed56f267401c9c501ad38c2cad1c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:14:07.806628 25615 raft_consensus.cc:3037] T 9597a33811ef4247a4c0ecb76d0afbaf P 9666ed56f267401c9c501ad38c2cad1c [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:14:07.809092 25615 raft_consensus.cc:504] T 9597a33811ef4247a4c0ecb76d0afbaf P 9666ed56f267401c9c501ad38c2cad1c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "9666ed56f267401c9c501ad38c2cad1c" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 43735 } }
I20240502 14:14:07.809820 25615 leader_election.cc:288] T 9597a33811ef4247a4c0ecb76d0afbaf P 9666ed56f267401c9c501ad38c2cad1c [CANDIDATE]: Term 2 election: Requested vote from peers 61c05b91821b450f9853ce940d18372b (127.19.94.194:40163)
I20240502 14:14:07.810624 21995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9597a33811ef4247a4c0ecb76d0afbaf" candidate_uuid: "9666ed56f267401c9c501ad38c2cad1c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "61c05b91821b450f9853ce940d18372b"
I20240502 14:14:07.810868 21995 raft_consensus.cc:3032] T 9597a33811ef4247a4c0ecb76d0afbaf P 61c05b91821b450f9853ce940d18372b [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:14:07.810968 21995 raft_consensus.cc:729] T 9597a33811ef4247a4c0ecb76d0afbaf P 61c05b91821b450f9853ce940d18372b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 61c05b91821b450f9853ce940d18372b, State: Running, Role: LEADER
I20240502 14:14:07.811270 21995 consensus_queue.cc:260] T 9597a33811ef4247a4c0ecb76d0afbaf P 61c05b91821b450f9853ce940d18372b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "9666ed56f267401c9c501ad38c2cad1c" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 43735 } }
I20240502 14:14:07.811548 21995 raft_consensus.cc:3037] T 9597a33811ef4247a4c0ecb76d0afbaf P 61c05b91821b450f9853ce940d18372b [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:14:07.814189 21995 raft_consensus.cc:2454] T 9597a33811ef4247a4c0ecb76d0afbaf P 61c05b91821b450f9853ce940d18372b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9666ed56f267401c9c501ad38c2cad1c in term 2.
I20240502 14:14:07.814755 25550 leader_election.cc:302] T 9597a33811ef4247a4c0ecb76d0afbaf P 9666ed56f267401c9c501ad38c2cad1c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 61c05b91821b450f9853ce940d18372b, 9666ed56f267401c9c501ad38c2cad1c; no voters:
I20240502 14:14:07.815089 25814 raft_consensus.cc:2781] T 9597a33811ef4247a4c0ecb76d0afbaf P 9666ed56f267401c9c501ad38c2cad1c [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:14:07.815261 25814 raft_consensus.cc:686] T 9597a33811ef4247a4c0ecb76d0afbaf P 9666ed56f267401c9c501ad38c2cad1c [term 2 LEADER]: Becoming Leader. State: Replica: 9666ed56f267401c9c501ad38c2cad1c, State: Running, Role: LEADER
I20240502 14:14:07.815526 25814 consensus_queue.cc:237] T 9597a33811ef4247a4c0ecb76d0afbaf P 9666ed56f267401c9c501ad38c2cad1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "9666ed56f267401c9c501ad38c2cad1c" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 43735 } }
I20240502 14:14:07.818239 21722 catalog_manager.cc:5484] T 9597a33811ef4247a4c0ecb76d0afbaf P 9666ed56f267401c9c501ad38c2cad1c reported cstate change: term changed from 1 to 2, leader changed from 61c05b91821b450f9853ce940d18372b (127.19.94.194) to 9666ed56f267401c9c501ad38c2cad1c (127.19.94.200). New cstate: current_term: 2 leader_uuid: "9666ed56f267401c9c501ad38c2cad1c" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 } } peers { permanent_uuid: "9666ed56f267401c9c501ad38c2cad1c" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 43735 } } }
I20240502 14:14:07.882925 21974 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:07.896129 23611 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:07.899106 21844 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:07.902549 25595 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:07.931759 23210 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:07.944082 23481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:07.945391 23341 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:07.967859 23078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:08.007994 21863 raft_consensus.cc:1261] T 7253bd03818a4c2f9f0478aebb326273 P 0c1d247517e246c2828007ca0a6d5527 [term 2 FOLLOWER]: Refusing update from remote peer 9666ed56f267401c9c501ad38c2cad1c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:14:08.009130 25814 consensus_queue.cc:1035] T 7253bd03818a4c2f9f0478aebb326273 P 9666ed56f267401c9c501ad38c2cad1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c1d247517e246c2828007ca0a6d5527" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 38827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:08.354563 21995 raft_consensus.cc:1261] T 9597a33811ef4247a4c0ecb76d0afbaf P 61c05b91821b450f9853ce940d18372b [term 2 FOLLOWER]: Refusing update from remote peer 9666ed56f267401c9c501ad38c2cad1c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:14:08.355284 25814 consensus_queue.cc:1035] T 9597a33811ef4247a4c0ecb76d0afbaf P 9666ed56f267401c9c501ad38c2cad1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "61c05b91821b450f9853ce940d18372b" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 40163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:08.393644 21844 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:08.413849 23611 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:08.419636 23341 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:08.435346 25595 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:08.476382 21974 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:08.492973 23481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:08.514534 23078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:08.521569 23210 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:08.597349 25615 consensus_queue.cc:237] T 7253bd03818a4c2f9f0478aebb326273 P 9666ed56f267401c9c501ad38c2cad1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9666ed56f267401c9c501ad38c2cad1c" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 43735 } }
I20240502 14:14:08.598316 25908 raft_consensus.cc:2932] T 7253bd03818a4c2f9f0478aebb326273 P 9666ed56f267401c9c501ad38c2cad1c [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9666ed56f267401c9c501ad38c2cad1c" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 43735 } } }
I20240502 14:14:08.602764 21718 catalog_manager.cc:5484] T 7253bd03818a4c2f9f0478aebb326273 P 9666ed56f267401c9c501ad38c2cad1c reported cstate change: config changed from index 4 to 6, VOTER 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193) evicted. New cstate: current_term: 2 leader_uuid: "9666ed56f267401c9c501ad38c2cad1c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9666ed56f267401c9c501ad38c2cad1c" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 43735 } } }
I20240502 14:14:08.608139 21844 tablet_service.cc:1514] Processing DeleteTablet for tablet 7253bd03818a4c2f9f0478aebb326273 with delete_type TABLET_DATA_TOMBSTONED (TS 0c1d247517e246c2828007ca0a6d5527 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44122
I20240502 14:14:08.608860 26072 tablet_replica.cc:330] T 7253bd03818a4c2f9f0478aebb326273 P 0c1d247517e246c2828007ca0a6d5527: stopping tablet replica
I20240502 14:14:08.609087 26072 raft_consensus.cc:2229] T 7253bd03818a4c2f9f0478aebb326273 P 0c1d247517e246c2828007ca0a6d5527 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:08.609300 26072 raft_consensus.cc:2258] T 7253bd03818a4c2f9f0478aebb326273 P 0c1d247517e246c2828007ca0a6d5527 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:08.611404 26072 ts_tablet_manager.cc:1875] T 7253bd03818a4c2f9f0478aebb326273 P 0c1d247517e246c2828007ca0a6d5527: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:14:08.616988 26072 ts_tablet_manager.cc:1888] T 7253bd03818a4c2f9f0478aebb326273 P 0c1d247517e246c2828007ca0a6d5527: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 14:14:08.617156 26072 log.cc:1196] T 7253bd03818a4c2f9f0478aebb326273 P 0c1d247517e246c2828007ca0a6d5527: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/7253bd03818a4c2f9f0478aebb326273
I20240502 14:14:08.618103 21708 catalog_manager.cc:4830] TS 0c1d247517e246c2828007ca0a6d5527 (127.19.94.193:38827): tablet 7253bd03818a4c2f9f0478aebb326273 (table rebalance_test_table_2 [id=1f28b28258954e02b4cf5c905ea2a48c]) successfully deleted
I20240502 14:14:08.729993 21844 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:08.736114 23611 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:08.748178 23341 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:08.749159 21974 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:08.754175 25595 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:08.791551 23210 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:08.796038 23078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:08.800990 23481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:08.890296 25615 consensus_queue.cc:237] T 9597a33811ef4247a4c0ecb76d0afbaf P 9666ed56f267401c9c501ad38c2cad1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9666ed56f267401c9c501ad38c2cad1c" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 43735 } }
I20240502 14:14:08.891382 25908 raft_consensus.cc:2932] T 9597a33811ef4247a4c0ecb76d0afbaf P 9666ed56f267401c9c501ad38c2cad1c [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 61c05b91821b450f9853ce940d18372b (127.19.94.194) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9666ed56f267401c9c501ad38c2cad1c" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 43735 } } }
I20240502 14:14:08.896236 21722 catalog_manager.cc:5484] T 9597a33811ef4247a4c0ecb76d0afbaf P 9666ed56f267401c9c501ad38c2cad1c reported cstate change: config changed from index 4 to 6, VOTER 61c05b91821b450f9853ce940d18372b (127.19.94.194) evicted. New cstate: current_term: 2 leader_uuid: "9666ed56f267401c9c501ad38c2cad1c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9666ed56f267401c9c501ad38c2cad1c" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 43735 } } }
I20240502 14:14:08.900507 21974 tablet_service.cc:1514] Processing DeleteTablet for tablet 9597a33811ef4247a4c0ecb76d0afbaf with delete_type TABLET_DATA_TOMBSTONED (TS 61c05b91821b450f9853ce940d18372b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46584
I20240502 14:14:08.901134 26120 tablet_replica.cc:330] T 9597a33811ef4247a4c0ecb76d0afbaf P 61c05b91821b450f9853ce940d18372b: stopping tablet replica
I20240502 14:14:08.901319 26120 raft_consensus.cc:2229] T 9597a33811ef4247a4c0ecb76d0afbaf P 61c05b91821b450f9853ce940d18372b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:08.901549 26120 raft_consensus.cc:2258] T 9597a33811ef4247a4c0ecb76d0afbaf P 61c05b91821b450f9853ce940d18372b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:08.903426 26120 ts_tablet_manager.cc:1875] T 9597a33811ef4247a4c0ecb76d0afbaf P 61c05b91821b450f9853ce940d18372b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:14:08.907933 26120 ts_tablet_manager.cc:1888] T 9597a33811ef4247a4c0ecb76d0afbaf P 61c05b91821b450f9853ce940d18372b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 14:14:08.908054 26120 log.cc:1196] T 9597a33811ef4247a4c0ecb76d0afbaf P 61c05b91821b450f9853ce940d18372b: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/9597a33811ef4247a4c0ecb76d0afbaf
I20240502 14:14:08.909051 21709 catalog_manager.cc:4830] TS 61c05b91821b450f9853ce940d18372b (127.19.94.194:40163): tablet 9597a33811ef4247a4c0ecb76d0afbaf (table rebalance_test_table_4 [id=fe06048058fb45dcb952409c2f9a3754]) successfully deleted
I20240502 14:14:09.240789 23611 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:09.255750 21974 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:09.264688 23341 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:09.265488 21844 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:09.295159 23481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:09.296353 23210 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:09.300783 23078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:09.310894 25595 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:09.517473 21974 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:09.526520 23611 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:09.535210 21844 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:09.544478 23481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:09.553560 23341 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:09.582289 23210 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:09.587553 23078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:09.589959 25595 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:09.932006 21844 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:09.934046 23611 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:09.946703 23341 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:09.947361 21974 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:09.968127 25595 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:10.001760 23481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:10.006402 23078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:10.025431 23210 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
66ce354883f14c8392e7fc8e9383bc85 | 127.19.94.254:40967 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.19.94.212:39595 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
0c1d247517e246c2828007ca0a6d5527 | 127.19.94.193:38827 | HEALTHY | <none> | 2 | 0
5ab00105ba8d4412bfb9d009c4e40352 | 127.19.94.199:44599 | HEALTHY | <none> | 2 | 0
61c05b91821b450f9853ce940d18372b | 127.19.94.194:40163 | HEALTHY | <none> | 2 | 0
92646999d5c64d1aae6f165593b7d14d | 127.19.94.197:33817 | HEALTHY | <none> | 2 | 0
9666ed56f267401c9c501ad38c2cad1c | 127.19.94.200:43735 | HEALTHY | <none> | 2 | 0
d53de47e3dbd4a37bb81b478ae4d1e0e | 127.19.94.198:41717 | HEALTHY | <none> | 2 | 0
f8fbf60396bb443697ce265de0eb063e | 127.19.94.196:41835 | HEALTHY | <none> | 2 | 0
ff15cd84fa6640b8921bbf05dc65612e | 127.19.94.195:39713 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.19.94.193 | experimental | 127.19.94.193:38827
local_ip_for_outbound_sockets | 127.19.94.194 | experimental | 127.19.94.194:40163
local_ip_for_outbound_sockets | 127.19.94.195 | experimental | 127.19.94.195:39713
local_ip_for_outbound_sockets | 127.19.94.196 | experimental | 127.19.94.196:41835
local_ip_for_outbound_sockets | 127.19.94.197 | experimental | 127.19.94.197:33817
local_ip_for_outbound_sockets | 127.19.94.198 | experimental | 127.19.94.198:41717
local_ip_for_outbound_sockets | 127.19.94.199 | experimental | 127.19.94.199:44599
local_ip_for_outbound_sockets | 127.19.94.200 | experimental | 127.19.94.200:43735
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.19.94.193:38827
server_dump_info_path | /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.94.194:40163
server_dump_info_path | /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.94.195:39713
server_dump_info_path | /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.19.94.196:41835
server_dump_info_path | /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.94.197:33817
server_dump_info_path | /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.94.198:41717
server_dump_info_path | /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.19.94.199:44599
server_dump_info_path | /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.19.94.200:43735
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.19.94.212:39595 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 2
Maximum | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 14:14:10.084978 19835 log_verifier.cc:126] Checking tablet 148d828a22194299a04c2e93003bf379
I20240502 14:14:10.095875 19835 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 148d828a22194299a04c2e93003bf379
I20240502 14:14:10.095984 19835 log_verifier.cc:126] Checking tablet 159f343b034a47e88f5d8ecfa0bc282c
I20240502 14:14:10.106062 19835 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 159f343b034a47e88f5d8ecfa0bc282c
I20240502 14:14:10.106159 19835 log_verifier.cc:126] Checking tablet 2ffa22a372c143bc87194122a407425c
I20240502 14:14:10.116187 19835 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 2ffa22a372c143bc87194122a407425c
I20240502 14:14:10.116286 19835 log_verifier.cc:126] Checking tablet 41b085facc334a7da0c7dd937275009f
I20240502 14:14:10.121646 19835 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 41b085facc334a7da0c7dd937275009f
I20240502 14:14:10.121740 19835 log_verifier.cc:126] Checking tablet 424a2170ba374ed1982e550e8f7ebeea
I20240502 14:14:10.131774 19835 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 424a2170ba374ed1982e550e8f7ebeea
I20240502 14:14:10.131875 19835 log_verifier.cc:126] Checking tablet 48dfb482c48d408aa359c0b948237fd6
I20240502 14:14:10.142053 19835 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 48dfb482c48d408aa359c0b948237fd6
I20240502 14:14:10.142153 19835 log_verifier.cc:126] Checking tablet 4cc5c4909e7343578a87c1b91ed3d756
I20240502 14:14:10.152525 19835 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 4cc5c4909e7343578a87c1b91ed3d756
I20240502 14:14:10.152657 19835 log_verifier.cc:126] Checking tablet 54ec5887ceee418c9915a1f043978f83
I20240502 14:14:10.162986 19835 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 54ec5887ceee418c9915a1f043978f83
I20240502 14:14:10.163097 19835 log_verifier.cc:126] Checking tablet 7253bd03818a4c2f9f0478aebb326273
I20240502 14:14:10.176577 19835 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 7253bd03818a4c2f9f0478aebb326273
I20240502 14:14:10.176687 19835 log_verifier.cc:126] Checking tablet 80edabe29f114dd2b3b958fc954e3c2b
I20240502 14:14:10.187848 19835 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 80edabe29f114dd2b3b958fc954e3c2b
I20240502 14:14:10.187953 19835 log_verifier.cc:126] Checking tablet 8a2d3ba227dd43eb941b7e1b3e8445ce
I20240502 14:14:10.197589 19835 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 8a2d3ba227dd43eb941b7e1b3e8445ce
I20240502 14:14:10.197687 19835 log_verifier.cc:126] Checking tablet 8af0e06b069e424a9ace44e06f12a6d3
I20240502 14:14:10.208773 19835 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 8af0e06b069e424a9ace44e06f12a6d3
I20240502 14:14:10.208894 19835 log_verifier.cc:126] Checking tablet 906a106c412944e4a049f1491999ad37
I20240502 14:14:10.219231 19835 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 906a106c412944e4a049f1491999ad37
I20240502 14:14:10.219336 19835 log_verifier.cc:126] Checking tablet 9597a33811ef4247a4c0ecb76d0afbaf
I20240502 14:14:10.233757 19835 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 9597a33811ef4247a4c0ecb76d0afbaf
I20240502 14:14:10.233888 19835 log_verifier.cc:126] Checking tablet ab54535b58be4e518340f33449bcdb26
I20240502 14:14:10.245947 19835 log_verifier.cc:177] Verified matching terms for 3 ops in tablet ab54535b58be4e518340f33449bcdb26
I20240502 14:14:10.246063 19835 log_verifier.cc:126] Checking tablet f3119216db7d4c0ea2216744e6f21c5f
I20240502 14:14:10.261479 19835 log_verifier.cc:177] Verified matching terms for 6 ops in tablet f3119216db7d4c0ea2216744e6f21c5f
I20240502 14:14:10.262466 19835 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_zeJNl/build/asan/bin/kudu pid 21782
I20240502 14:14:10.264045 21904 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:14:10.490037 19835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_zeJNl/build/asan/bin/kudu with pid 21782
I20240502 14:14:10.530851 19835 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_zeJNl/build/asan/bin/kudu pid 21913
I20240502 14:14:10.532109 22035 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:14:10.759048 19835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_zeJNl/build/asan/bin/kudu with pid 21913
I20240502 14:14:10.796367 19835 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_zeJNl/build/asan/bin/kudu pid 23016
I20240502 14:14:10.804519 23138 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:14:11.013234 19835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_zeJNl/build/asan/bin/kudu with pid 23016
I20240502 14:14:11.047691 19835 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_zeJNl/build/asan/bin/kudu pid 23147
I20240502 14:14:11.055076 23269 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:14:11.245599 19835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_zeJNl/build/asan/bin/kudu with pid 23147
I20240502 14:14:11.277765 19835 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_zeJNl/build/asan/bin/kudu pid 23278
I20240502 14:14:11.284363 23401 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:14:11.473258 19835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_zeJNl/build/asan/bin/kudu with pid 23278
I20240502 14:14:11.505188 19835 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_zeJNl/build/asan/bin/kudu pid 23413
I20240502 14:14:11.512370 23540 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:14:11.700996 19835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_zeJNl/build/asan/bin/kudu with pid 23413
I20240502 14:14:11.732863 19835 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_zeJNl/build/asan/bin/kudu pid 23549
I20240502 14:14:11.740231 23670 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:14:11.942950 19835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_zeJNl/build/asan/bin/kudu with pid 23549
I20240502 14:14:11.977962 19835 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_zeJNl/build/asan/bin/kudu pid 25533
I20240502 14:14:11.986518 25655 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:14:12.106477 19835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_zeJNl/build/asan/bin/kudu with pid 25533
I20240502 14:14:12.129853 19835 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_zeJNl/build/asan/bin/kudu pid 21692
I20240502 14:14:12.131125 21753 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:14:12.385630 19835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_zeJNl/build/asan/bin/kudu with pid 21692
2024-05-02T14:14:12Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/0 (44987 ms)
[----------] 1 test from RebalanceParamTest (44987 ms total)
[----------] 1 test from DDLDuringRebalancingTest
[ RUN ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/0
I20240502 14:14:12.477615 19835 test_util.cc:268] Using random seed: 933731991
I20240502 14:14:12.479825 19835 ts_itest-base.cc:112] Starting cluster with:
I20240502 14:14:12.479895 19835 ts_itest-base.cc:113] --------------
I20240502 14:14:12.479940 19835 ts_itest-base.cc:114] 8 tablet servers
I20240502 14:14:12.479985 19835 ts_itest-base.cc:115] 3 replicas per TS
I20240502 14:14:12.480031 19835 ts_itest-base.cc:116] --------------
2024-05-02T14:14:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:14:12Z Disabled control of system clock
I20240502 14:14:12.516989 19835 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_zeJNl/build/asan/bin/kudu
/tmp/dist-test-task_zeJNl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.94.254:34367
--webserver_interface=127.19.94.254
--webserver_port=0
--builtin_ntp_servers=127.19.94.212:40605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.94.254:34367
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240502 14:14:12.886059 26254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:12.886421 26254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:14:12.886554 26254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:12.895114 26254 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:14:12.895289 26254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:14:12.895527 26254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:14:12.895654 26254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:14:12.895728 26254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:14:12.906637 26254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.94.212:40605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.19.94.254:34367
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.94.254:34367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.19.94.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:14:12.908181 26254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:14:12.910156 26254 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:12.923842 26260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:12.924679 26254 server_base.cc:1034] running on GCE node
W20240502 14:14:12.925463 26262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:12.926481 26259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:12.927584 26254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:14:12.929548 26254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:14:12.930856 26254 hybrid_clock.cc:648] HybridClock initialized: now 1714659252930745 us; error 101 us; skew 500 ppm
I20240502 14:14:12.931308 26254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:12.941051 26254 webserver.cc:460] Webserver started at http://127.19.94.254:33983/ using document root <none> and password file <none>
I20240502 14:14:12.941998 26254 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:12.942142 26254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:12.942499 26254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:12.945533 26254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "09bae9f1528d44c09ade01fa361cebd0"
format_stamp: "Formatted at 2024-05-02 14:14:12 on dist-test-slave-3j2v"
I20240502 14:14:12.946398 26254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "09bae9f1528d44c09ade01fa361cebd0"
format_stamp: "Formatted at 2024-05-02 14:14:12 on dist-test-slave-3j2v"
I20240502 14:14:12.952306 26254 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20240502 14:14:12.956182 26268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:12.958217 26254 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240502 14:14:12.958441 26254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/wal
uuid: "09bae9f1528d44c09ade01fa361cebd0"
format_stamp: "Formatted at 2024-05-02 14:14:12 on dist-test-slave-3j2v"
I20240502 14:14:12.958689 26254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:12.991735 26254 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:12.992722 26254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:14:12.993022 26254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:13.014315 26254 rpc_server.cc:307] RPC server started. Bound to: 127.19.94.254:34367
I20240502 14:14:13.014357 26319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.94.254:34367 every 8 connection(s)
I20240502 14:14:13.015890 26254 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 14:14:13.020080 26320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:13.023941 19835 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_zeJNl/build/asan/bin/kudu as pid 26254
I20240502 14:14:13.024190 19835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 14:14:13.033478 26320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Bootstrap starting.
I20240502 14:14:13.037338 26320 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:13.038580 26320 log.cc:824] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Log is configured to *not* fsync() on all Append() calls
I20240502 14:14:13.042455 26320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: No bootstrap required, opened a new log
I20240502 14:14:13.049459 26320 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09bae9f1528d44c09ade01fa361cebd0" member_type: VOTER last_known_addr { host: "127.19.94.254" port: 34367 } }
I20240502 14:14:13.049875 26320 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:13.049993 26320 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09bae9f1528d44c09ade01fa361cebd0, State: Initialized, Role: FOLLOWER
I20240502 14:14:13.050702 26320 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09bae9f1528d44c09ade01fa361cebd0" member_type: VOTER last_known_addr { host: "127.19.94.254" port: 34367 } }
I20240502 14:14:13.050921 26320 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:14:13.051061 26320 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:14:13.051237 26320 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:13.054442 26320 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09bae9f1528d44c09ade01fa361cebd0" member_type: VOTER last_known_addr { host: "127.19.94.254" port: 34367 } }
I20240502 14:14:13.055053 26320 leader_election.cc:302] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09bae9f1528d44c09ade01fa361cebd0; no voters:
I20240502 14:14:13.055691 26320 leader_election.cc:288] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:14:13.055835 26325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:13.056627 26325 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0 [term 1 LEADER]: Becoming Leader. State: Replica: 09bae9f1528d44c09ade01fa361cebd0, State: Running, Role: LEADER
I20240502 14:14:13.057619 26325 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09bae9f1528d44c09ade01fa361cebd0" member_type: VOTER last_known_addr { host: "127.19.94.254" port: 34367 } }
I20240502 14:14:13.058071 26320 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:14:13.061201 26327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09bae9f1528d44c09ade01fa361cebd0. Latest consensus state: current_term: 1 leader_uuid: "09bae9f1528d44c09ade01fa361cebd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09bae9f1528d44c09ade01fa361cebd0" member_type: VOTER last_known_addr { host: "127.19.94.254" port: 34367 } } }
I20240502 14:14:13.061595 26327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0 [sys.catalog]: This master's current role is: LEADER
I20240502 14:14:13.062139 26326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09bae9f1528d44c09ade01fa361cebd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09bae9f1528d44c09ade01fa361cebd0" member_type: VOTER last_known_addr { host: "127.19.94.254" port: 34367 } } }
I20240502 14:14:13.062456 26326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0 [sys.catalog]: This master's current role is: LEADER
I20240502 14:14:13.066246 26333 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:14:13.071756 26333 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:14:13.082065 26333 catalog_manager.cc:1347] Generated new cluster ID: 1436728982514b96acad76ab33973fe6
I20240502 14:14:13.082262 26333 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:14:13.115933 26333 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:14:13.117853 26333 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:14:13.128907 26333 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Generated new TSK 0
I20240502 14:14:13.130360 26333 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:14:13.145327 19835 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_zeJNl/build/asan/bin/kudu
/tmp/dist-test-task_zeJNl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.94.193:0
--local_ip_for_outbound_sockets=127.19.94.193
--webserver_interface=127.19.94.193
--webserver_port=0
--tserver_master_addrs=127.19.94.254:34367
--builtin_ntp_servers=127.19.94.212:40605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 14:14:13.553378 26345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:13.553707 26345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:14:13.553807 26345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:13.561758 26345 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:14:13.561906 26345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:14:13.562062 26345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.94.193
I20240502 14:14:13.572470 26345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.94.212:40605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.94.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.19.94.193
--webserver_port=0
--tserver_master_addrs=127.19.94.254:34367
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.94.193
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:14:13.573936 26345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:14:13.575852 26345 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:13.590416 26350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:13.590983 26351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:13.591014 26353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:13.591662 26345 server_base.cc:1034] running on GCE node
I20240502 14:14:13.592300 26345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:14:13.593358 26345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:14:13.594578 26345 hybrid_clock.cc:648] HybridClock initialized: now 1714659253594518 us; error 40 us; skew 500 ppm
I20240502 14:14:13.594900 26345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:13.597282 26345 webserver.cc:460] Webserver started at http://127.19.94.193:43473/ using document root <none> and password file <none>
I20240502 14:14:13.598053 26345 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:13.598176 26345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:13.598503 26345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:13.601390 26345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "89af7f423cf44717a08d6b142f0ed49d"
format_stamp: "Formatted at 2024-05-02 14:14:13 on dist-test-slave-3j2v"
I20240502 14:14:13.602303 26345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "89af7f423cf44717a08d6b142f0ed49d"
format_stamp: "Formatted at 2024-05-02 14:14:13 on dist-test-slave-3j2v"
I20240502 14:14:13.608263 26345 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20240502 14:14:13.613309 26359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:13.615085 26345 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20240502 14:14:13.615286 26345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "89af7f423cf44717a08d6b142f0ed49d"
format_stamp: "Formatted at 2024-05-02 14:14:13 on dist-test-slave-3j2v"
I20240502 14:14:13.615515 26345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:13.640980 26345 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:13.641995 26345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:14:13.642306 26345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:13.643537 26345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:13.645387 26345 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:13.645500 26345 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:13.645628 26345 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:13.645707 26345 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:13.683272 26345 rpc_server.cc:307] RPC server started. Bound to: 127.19.94.193:37985
I20240502 14:14:13.683326 26471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.94.193:37985 every 8 connection(s)
I20240502 14:14:13.684674 26345 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 14:14:13.685698 19835 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_zeJNl/build/asan/bin/kudu as pid 26345
I20240502 14:14:13.685936 19835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 14:14:13.689783 19835 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_zeJNl/build/asan/bin/kudu
/tmp/dist-test-task_zeJNl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.94.194:0
--local_ip_for_outbound_sockets=127.19.94.194
--webserver_interface=127.19.94.194
--webserver_port=0
--tserver_master_addrs=127.19.94.254:34367
--builtin_ntp_servers=127.19.94.212:40605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:14:13.701298 26472 heartbeater.cc:347] Connected to a master server at 127.19.94.254:34367
I20240502 14:14:13.701650 26472 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:13.702543 26472 heartbeater.cc:511] Master 127.19.94.254:34367 requested a full tablet report, sending...
I20240502 14:14:13.704879 26285 ts_manager.cc:194] Registered new tserver with Master: 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:14:13.706581 26285 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.94.193:44455
W20240502 14:14:14.046304 26476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:14.046644 26476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:14:14.046784 26476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:14.054831 26476 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:14:14.055011 26476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:14:14.055188 26476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.94.194
I20240502 14:14:14.065411 26476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.94.212:40605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.94.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.19.94.194
--webserver_port=0
--tserver_master_addrs=127.19.94.254:34367
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.94.194
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:14:14.066884 26476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:14:14.068800 26476 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:14.082633 26481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:14.083257 26482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:14.084071 26484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:14.084936 26476 server_base.cc:1034] running on GCE node
I20240502 14:14:14.085611 26476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:14:14.086633 26476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:14:14.087872 26476 hybrid_clock.cc:648] HybridClock initialized: now 1714659254087816 us; error 50 us; skew 500 ppm
I20240502 14:14:14.088250 26476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:14.095479 26476 webserver.cc:460] Webserver started at http://127.19.94.194:36045/ using document root <none> and password file <none>
I20240502 14:14:14.096432 26476 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:14.096565 26476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:14.096915 26476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:14.099792 26476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "ba187e38a2754d60a12de9ac4856430a"
format_stamp: "Formatted at 2024-05-02 14:14:14 on dist-test-slave-3j2v"
I20240502 14:14:14.100818 26476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "ba187e38a2754d60a12de9ac4856430a"
format_stamp: "Formatted at 2024-05-02 14:14:14 on dist-test-slave-3j2v"
I20240502 14:14:14.108079 26476 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20240502 14:14:14.113878 26490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:14.116230 26476 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20240502 14:14:14.116544 26476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ba187e38a2754d60a12de9ac4856430a"
format_stamp: "Formatted at 2024-05-02 14:14:14 on dist-test-slave-3j2v"
I20240502 14:14:14.116874 26476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:14.150296 26476 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:14.151670 26476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:14:14.152082 26476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:14.153677 26476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:14.156209 26476 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:14.156358 26476 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:14.156534 26476 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:14.156639 26476 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:14.198766 26476 rpc_server.cc:307] RPC server started. Bound to: 127.19.94.194:36025
I20240502 14:14:14.198844 26602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.94.194:36025 every 8 connection(s)
I20240502 14:14:14.200280 26476 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 14:14:14.206943 19835 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_zeJNl/build/asan/bin/kudu as pid 26476
I20240502 14:14:14.207120 19835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 14:14:14.210566 19835 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_zeJNl/build/asan/bin/kudu
/tmp/dist-test-task_zeJNl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.94.195:0
--local_ip_for_outbound_sockets=127.19.94.195
--webserver_interface=127.19.94.195
--webserver_port=0
--tserver_master_addrs=127.19.94.254:34367
--builtin_ntp_servers=127.19.94.212:40605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:14:14.213724 26603 heartbeater.cc:347] Connected to a master server at 127.19.94.254:34367
I20240502 14:14:14.214066 26603 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:14.214864 26603 heartbeater.cc:511] Master 127.19.94.254:34367 requested a full tablet report, sending...
I20240502 14:14:14.216420 26285 ts_manager.cc:194] Registered new tserver with Master: ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:14:14.217240 26285 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.94.194:59605
W20240502 14:14:14.572336 26607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:14.572733 26607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:14:14.572883 26607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:14.580763 26607 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:14:14.580931 26607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:14:14.581104 26607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.94.195
I20240502 14:14:14.590826 26607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.94.212:40605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.94.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.19.94.195
--webserver_port=0
--tserver_master_addrs=127.19.94.254:34367
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.94.195
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:14:14.592242 26607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:14:14.594033 26607 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:14.607738 26612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:14.608503 26613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:14.609241 26615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:14.610008 26607 server_base.cc:1034] running on GCE node
I20240502 14:14:14.610720 26607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:14:14.611811 26607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:14:14.613035 26607 hybrid_clock.cc:648] HybridClock initialized: now 1714659254612981 us; error 53 us; skew 500 ppm
I20240502 14:14:14.613415 26607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:14.616210 26607 webserver.cc:460] Webserver started at http://127.19.94.195:38487/ using document root <none> and password file <none>
I20240502 14:14:14.617249 26607 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:14.617403 26607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:14.617758 26607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:14.621193 26607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
format_stamp: "Formatted at 2024-05-02 14:14:14 on dist-test-slave-3j2v"
I20240502 14:14:14.622121 26607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
format_stamp: "Formatted at 2024-05-02 14:14:14 on dist-test-slave-3j2v"
I20240502 14:14:14.628232 26607 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20240502 14:14:14.632022 26621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:14.633711 26607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:14:14.633900 26607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
format_stamp: "Formatted at 2024-05-02 14:14:14 on dist-test-slave-3j2v"
I20240502 14:14:14.634119 26607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:14.654608 26607 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:14.655586 26607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:14:14.655902 26607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:14.657147 26607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:14.658959 26607 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:14.659057 26607 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:14.659163 26607 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:14.659222 26607 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:14.692633 26607 rpc_server.cc:307] RPC server started. Bound to: 127.19.94.195:44325
I20240502 14:14:14.692682 26733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.94.195:44325 every 8 connection(s)
I20240502 14:14:14.693928 26607 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 14:14:14.698323 19835 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_zeJNl/build/asan/bin/kudu as pid 26607
I20240502 14:14:14.698522 19835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 14:14:14.702643 19835 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_zeJNl/build/asan/bin/kudu
/tmp/dist-test-task_zeJNl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.94.196:0
--local_ip_for_outbound_sockets=127.19.94.196
--webserver_interface=127.19.94.196
--webserver_port=0
--tserver_master_addrs=127.19.94.254:34367
--builtin_ntp_servers=127.19.94.212:40605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:14:14.708878 26734 heartbeater.cc:347] Connected to a master server at 127.19.94.254:34367
I20240502 14:14:14.709271 26734 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:14.709764 26472 heartbeater.cc:503] Master 127.19.94.254:34367 was elected leader, sending a full tablet report...
I20240502 14:14:14.710372 26734 heartbeater.cc:511] Master 127.19.94.254:34367 requested a full tablet report, sending...
I20240502 14:14:14.712268 26285 ts_manager.cc:194] Registered new tserver with Master: 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:14:14.713130 26285 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.94.195:34069
W20240502 14:14:15.065452 26737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:15.065773 26737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:14:15.065888 26737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:15.074018 26737 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:14:15.074177 26737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:14:15.074344 26737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.94.196
I20240502 14:14:15.083992 26737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.94.212:40605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.94.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.19.94.196
--webserver_port=0
--tserver_master_addrs=127.19.94.254:34367
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.94.196
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:14:15.085438 26737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:14:15.087244 26737 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:15.100320 26742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:15.100971 26743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:15.102216 26745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:15.103276 26737 server_base.cc:1034] running on GCE node
I20240502 14:14:15.103964 26737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:14:15.105286 26737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:14:15.106555 26737 hybrid_clock.cc:648] HybridClock initialized: now 1714659255106518 us; error 41 us; skew 500 ppm
I20240502 14:14:15.106891 26737 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:15.110307 26737 webserver.cc:460] Webserver started at http://127.19.94.196:35567/ using document root <none> and password file <none>
I20240502 14:14:15.111132 26737 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:15.111254 26737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:15.111547 26737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:15.114427 26737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "e4cfb8fc5a474177af6848a6efa13a27"
format_stamp: "Formatted at 2024-05-02 14:14:15 on dist-test-slave-3j2v"
I20240502 14:14:15.115425 26737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "e4cfb8fc5a474177af6848a6efa13a27"
format_stamp: "Formatted at 2024-05-02 14:14:15 on dist-test-slave-3j2v"
I20240502 14:14:15.123126 26737 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20240502 14:14:15.128427 26751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:15.130434 26737 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240502 14:14:15.130652 26737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "e4cfb8fc5a474177af6848a6efa13a27"
format_stamp: "Formatted at 2024-05-02 14:14:15 on dist-test-slave-3j2v"
I20240502 14:14:15.130911 26737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:15.173691 26737 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:15.174568 26737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:14:15.174827 26737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:15.175830 26737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:15.177476 26737 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:15.177573 26737 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:15.177670 26737 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:15.177726 26737 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:15.213024 26737 rpc_server.cc:307] RPC server started. Bound to: 127.19.94.196:32843
I20240502 14:14:15.213173 26863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.94.196:32843 every 8 connection(s)
I20240502 14:14:15.214617 26737 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 14:14:15.221259 26603 heartbeater.cc:503] Master 127.19.94.254:34367 was elected leader, sending a full tablet report...
I20240502 14:14:15.221598 19835 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_zeJNl/build/asan/bin/kudu as pid 26737
I20240502 14:14:15.221787 19835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 14:14:15.225339 19835 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_zeJNl/build/asan/bin/kudu
/tmp/dist-test-task_zeJNl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.94.197:0
--local_ip_for_outbound_sockets=127.19.94.197
--webserver_interface=127.19.94.197
--webserver_port=0
--tserver_master_addrs=127.19.94.254:34367
--builtin_ntp_servers=127.19.94.212:40605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:14:15.229245 26864 heartbeater.cc:347] Connected to a master server at 127.19.94.254:34367
I20240502 14:14:15.229590 26864 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:15.230365 26864 heartbeater.cc:511] Master 127.19.94.254:34367 requested a full tablet report, sending...
I20240502 14:14:15.231889 26285 ts_manager.cc:194] Registered new tserver with Master: e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:15.232791 26285 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.94.196:55333
W20240502 14:14:15.578513 26868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:15.578850 26868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:14:15.578989 26868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:15.586624 26868 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:14:15.586786 26868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:14:15.586953 26868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.94.197
I20240502 14:14:15.596335 26868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.94.212:40605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.94.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.19.94.197
--webserver_port=0
--tserver_master_addrs=127.19.94.254:34367
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.94.197
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:14:15.597748 26868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:14:15.599490 26868 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:15.614058 26874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:15.614897 26873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:15.615540 26876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:15.616451 26868 server_base.cc:1034] running on GCE node
I20240502 14:14:15.617108 26868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:14:15.618186 26868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:14:15.619546 26868 hybrid_clock.cc:648] HybridClock initialized: now 1714659255619451 us; error 80 us; skew 500 ppm
I20240502 14:14:15.619930 26868 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:15.623210 26868 webserver.cc:460] Webserver started at http://127.19.94.197:44271/ using document root <none> and password file <none>
I20240502 14:14:15.624081 26868 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:15.624197 26868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:15.624712 26868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:15.627741 26868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "1cb67158133e4648a6eceba32d4dbba3"
format_stamp: "Formatted at 2024-05-02 14:14:15 on dist-test-slave-3j2v"
I20240502 14:14:15.628648 26868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "1cb67158133e4648a6eceba32d4dbba3"
format_stamp: "Formatted at 2024-05-02 14:14:15 on dist-test-slave-3j2v"
I20240502 14:14:15.634313 26868 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 14:14:15.638197 26882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:15.640025 26868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240502 14:14:15.640332 26868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "1cb67158133e4648a6eceba32d4dbba3"
format_stamp: "Formatted at 2024-05-02 14:14:15 on dist-test-slave-3j2v"
I20240502 14:14:15.640601 26868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:15.664814 26868 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:15.665755 26868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:14:15.666041 26868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:15.667171 26868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:15.668761 26868 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:15.668874 26868 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:15.668999 26868 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:15.669092 26868 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:15.704200 26868 rpc_server.cc:307] RPC server started. Bound to: 127.19.94.197:38715
I20240502 14:14:15.704253 26994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.94.197:38715 every 8 connection(s)
I20240502 14:14:15.705541 26868 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 14:14:15.711586 19835 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_zeJNl/build/asan/bin/kudu as pid 26868
I20240502 14:14:15.711802 19835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 14:14:15.715365 19835 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_zeJNl/build/asan/bin/kudu
/tmp/dist-test-task_zeJNl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.94.198:0
--local_ip_for_outbound_sockets=127.19.94.198
--webserver_interface=127.19.94.198
--webserver_port=0
--tserver_master_addrs=127.19.94.254:34367
--builtin_ntp_servers=127.19.94.212:40605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:14:15.716449 26734 heartbeater.cc:503] Master 127.19.94.254:34367 was elected leader, sending a full tablet report...
I20240502 14:14:15.720058 26995 heartbeater.cc:347] Connected to a master server at 127.19.94.254:34367
I20240502 14:14:15.720433 26995 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:15.721316 26995 heartbeater.cc:511] Master 127.19.94.254:34367 requested a full tablet report, sending...
I20240502 14:14:15.722890 26285 ts_manager.cc:194] Registered new tserver with Master: 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:14:15.724155 26285 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.94.197:54137
W20240502 14:14:16.077394 26998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:16.077710 26998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:14:16.077843 26998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:16.085718 26998 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:14:16.085861 26998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:14:16.085984 26998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.94.198
I20240502 14:14:16.095291 26998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.94.212:40605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.94.198:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.19.94.198
--webserver_port=0
--tserver_master_addrs=127.19.94.254:34367
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.94.198
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:14:16.096661 26998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:14:16.098433 26998 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:16.112198 27004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:16.112783 27006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:16.113436 27003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:16.114182 26998 server_base.cc:1034] running on GCE node
I20240502 14:14:16.114847 26998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:14:16.115887 26998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:14:16.117112 26998 hybrid_clock.cc:648] HybridClock initialized: now 1714659256117045 us; error 56 us; skew 500 ppm
I20240502 14:14:16.117475 26998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:16.120149 26998 webserver.cc:460] Webserver started at http://127.19.94.198:46553/ using document root <none> and password file <none>
I20240502 14:14:16.121038 26998 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:16.121268 26998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:16.121651 26998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:16.124523 26998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
format_stamp: "Formatted at 2024-05-02 14:14:16 on dist-test-slave-3j2v"
I20240502 14:14:16.125382 26998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
format_stamp: "Formatted at 2024-05-02 14:14:16 on dist-test-slave-3j2v"
I20240502 14:14:16.131126 26998 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 14:14:16.134810 27012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:16.136492 26998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:14:16.136669 26998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
format_stamp: "Formatted at 2024-05-02 14:14:16 on dist-test-slave-3j2v"
I20240502 14:14:16.136905 26998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:16.187623 26998 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:16.188700 26998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:14:16.188994 26998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:16.190109 26998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:16.191838 26998 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:16.191931 26998 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:16.192013 26998 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:16.192068 26998 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:16.232373 26998 rpc_server.cc:307] RPC server started. Bound to: 127.19.94.198:43659
I20240502 14:14:16.232537 27124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.94.198:43659 every 8 connection(s)
I20240502 14:14:16.233774 26998 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 14:14:16.235944 26864 heartbeater.cc:503] Master 127.19.94.254:34367 was elected leader, sending a full tablet report...
I20240502 14:14:16.242360 19835 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_zeJNl/build/asan/bin/kudu as pid 26998
I20240502 14:14:16.242524 19835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 14:14:16.246806 19835 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_zeJNl/build/asan/bin/kudu
/tmp/dist-test-task_zeJNl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.94.199:0
--local_ip_for_outbound_sockets=127.19.94.199
--webserver_interface=127.19.94.199
--webserver_port=0
--tserver_master_addrs=127.19.94.254:34367
--builtin_ntp_servers=127.19.94.212:40605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:14:16.249167 27125 heartbeater.cc:347] Connected to a master server at 127.19.94.254:34367
I20240502 14:14:16.249575 27125 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:16.250643 27125 heartbeater.cc:511] Master 127.19.94.254:34367 requested a full tablet report, sending...
I20240502 14:14:16.253237 26285 ts_manager.cc:194] Registered new tserver with Master: c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:14:16.254436 26285 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.94.198:36033
W20240502 14:14:16.600915 27129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:16.601226 27129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:14:16.601336 27129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:16.609848 27129 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:14:16.610025 27129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:14:16.610257 27129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.94.199
I20240502 14:14:16.619889 27129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.94.212:40605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.94.199:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.19.94.199
--webserver_port=0
--tserver_master_addrs=127.19.94.254:34367
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.94.199
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:14:16.621402 27129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:14:16.623220 27129 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:16.636756 27134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:16.637423 27135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:16.638000 27137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:16.638756 27129 server_base.cc:1034] running on GCE node
I20240502 14:14:16.639501 27129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:14:16.640627 27129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:14:16.641867 27129 hybrid_clock.cc:648] HybridClock initialized: now 1714659256641815 us; error 45 us; skew 500 ppm
I20240502 14:14:16.642272 27129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:16.644855 27129 webserver.cc:460] Webserver started at http://127.19.94.199:44101/ using document root <none> and password file <none>
I20240502 14:14:16.645648 27129 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:16.645792 27129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:16.646245 27129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:16.649282 27129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "68a3c2b17ed94e06aba66ed4178132ab"
format_stamp: "Formatted at 2024-05-02 14:14:16 on dist-test-slave-3j2v"
I20240502 14:14:16.650241 27129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "68a3c2b17ed94e06aba66ed4178132ab"
format_stamp: "Formatted at 2024-05-02 14:14:16 on dist-test-slave-3j2v"
I20240502 14:14:16.655910 27129 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240502 14:14:16.659845 27143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:16.661721 27129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:14:16.661901 27129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "68a3c2b17ed94e06aba66ed4178132ab"
format_stamp: "Formatted at 2024-05-02 14:14:16 on dist-test-slave-3j2v"
I20240502 14:14:16.662148 27129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:16.681443 27129 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:16.682407 27129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:14:16.682731 27129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:16.683951 27129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:16.685637 27129 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:16.685732 27129 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:16.685856 27129 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:16.685930 27129 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:16.722049 27129 rpc_server.cc:307] RPC server started. Bound to: 127.19.94.199:41781
I20240502 14:14:16.722254 27255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.94.199:41781 every 8 connection(s)
I20240502 14:14:16.723402 27129 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 14:14:16.726821 26995 heartbeater.cc:503] Master 127.19.94.254:34367 was elected leader, sending a full tablet report...
I20240502 14:14:16.732957 19835 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_zeJNl/build/asan/bin/kudu as pid 27129
I20240502 14:14:16.733179 19835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 14:14:16.736568 27256 heartbeater.cc:347] Connected to a master server at 127.19.94.254:34367
I20240502 14:14:16.736891 27256 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:16.736954 19835 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_zeJNl/build/asan/bin/kudu
/tmp/dist-test-task_zeJNl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.94.200:0
--local_ip_for_outbound_sockets=127.19.94.200
--webserver_interface=127.19.94.200
--webserver_port=0
--tserver_master_addrs=127.19.94.254:34367
--builtin_ntp_servers=127.19.94.212:40605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:14:16.737691 27256 heartbeater.cc:511] Master 127.19.94.254:34367 requested a full tablet report, sending...
I20240502 14:14:16.739109 26285 ts_manager.cc:194] Registered new tserver with Master: 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:14:16.739955 26285 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.94.199:44533
W20240502 14:14:17.088474 27259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:17.088801 27259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:14:17.088945 27259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:17.097129 27259 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:14:17.097285 27259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:14:17.097457 27259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.94.200
I20240502 14:14:17.107640 27259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.94.212:40605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.94.200:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.19.94.200
--webserver_port=0
--tserver_master_addrs=127.19.94.254:34367
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.94.200
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:14:17.109259 27259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:14:17.111104 27259 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:17.125507 27265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:17.126047 27267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:17.126717 27264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:17.127593 27259 server_base.cc:1034] running on GCE node
I20240502 14:14:17.128378 27259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:14:17.129472 27259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:14:17.130693 27259 hybrid_clock.cc:648] HybridClock initialized: now 1714659257130627 us; error 51 us; skew 500 ppm
I20240502 14:14:17.131035 27259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:17.133662 27259 webserver.cc:460] Webserver started at http://127.19.94.200:37805/ using document root <none> and password file <none>
I20240502 14:14:17.134445 27259 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:17.134558 27259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:17.134892 27259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:17.137897 27259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "4092902b3e674857a0147f1b4f2cdb86"
format_stamp: "Formatted at 2024-05-02 14:14:17 on dist-test-slave-3j2v"
I20240502 14:14:17.138803 27259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "4092902b3e674857a0147f1b4f2cdb86"
format_stamp: "Formatted at 2024-05-02 14:14:17 on dist-test-slave-3j2v"
I20240502 14:14:17.145689 27259 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20240502 14:14:17.149935 27273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:17.151574 27259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20240502 14:14:17.151769 27259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "4092902b3e674857a0147f1b4f2cdb86"
format_stamp: "Formatted at 2024-05-02 14:14:17 on dist-test-slave-3j2v"
I20240502 14:14:17.151993 27259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:17.171694 27259 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:17.172772 27259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:14:17.173094 27259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:17.174124 27259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:17.175824 27259 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:17.175932 27259 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:17.176060 27259 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:17.176137 27259 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:17.212985 27259 rpc_server.cc:307] RPC server started. Bound to: 127.19.94.200:46097
I20240502 14:14:17.213025 27385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.94.200:46097 every 8 connection(s)
I20240502 14:14:17.214522 27259 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 14:14:17.221313 19835 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_zeJNl/build/asan/bin/kudu as pid 27259
I20240502 14:14:17.221505 19835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 14:14:17.230405 27386 heartbeater.cc:347] Connected to a master server at 127.19.94.254:34367
I20240502 14:14:17.230723 27386 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:17.231523 27386 heartbeater.cc:511] Master 127.19.94.254:34367 requested a full tablet report, sending...
I20240502 14:14:17.233307 26285 ts_manager.cc:194] Registered new tserver with Master: 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:14:17.234606 26285 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.94.200:53273
I20240502 14:14:17.239614 19835 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 14:14:17.257962 27125 heartbeater.cc:503] Master 127.19.94.254:34367 was elected leader, sending a full tablet report...
I20240502 14:14:17.258364 19835 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 14:14:17.258503 19835 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_zeJNl/build/asan/bin/kudu pid 26868
I20240502 14:14:17.267827 26990 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:14:17.350804 19835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_zeJNl/build/asan/bin/kudu with pid 26868
I20240502 14:14:17.368817 19835 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_zeJNl/build/asan/bin/kudu pid 26998
I20240502 14:14:17.378175 27120 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:14:17.462679 19835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_zeJNl/build/asan/bin/kudu with pid 26998
I20240502 14:14:17.480085 19835 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_zeJNl/build/asan/bin/kudu pid 27129
I20240502 14:14:17.488209 27251 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:14:17.572078 19835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_zeJNl/build/asan/bin/kudu with pid 27129
I20240502 14:14:17.589257 19835 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_zeJNl/build/asan/bin/kudu pid 27259
I20240502 14:14:17.598037 27381 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:14:17.680907 19835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_zeJNl/build/asan/bin/kudu with pid 27259
I20240502 14:14:21.450469 26284 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:21.494956 26799 tablet_service.cc:1467] Processing CreateTablet for tablet bbe54b53e86a48fbb1ba02074fb7afe3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=911ed82f4a3a4ad4938e4b1c8608e51b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:21.496057 26537 tablet_service.cc:1467] Processing CreateTablet for tablet 52061de1d76a420db5f3eb0aad70bf3a (DEFAULT_TABLE table=rebalance_test_table_0 [id=911ed82f4a3a4ad4938e4b1c8608e51b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:21.496789 26799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbe54b53e86a48fbb1ba02074fb7afe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:21.497507 26537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52061de1d76a420db5f3eb0aad70bf3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:21.497598 26798 tablet_service.cc:1467] Processing CreateTablet for tablet 52061de1d76a420db5f3eb0aad70bf3a (DEFAULT_TABLE table=rebalance_test_table_0 [id=911ed82f4a3a4ad4938e4b1c8608e51b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:21.498090 26798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52061de1d76a420db5f3eb0aad70bf3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:21.496078 26538 tablet_service.cc:1467] Processing CreateTablet for tablet 63c1903111784cf38265e2eeb4faff2e (DEFAULT_TABLE table=rebalance_test_table_0 [id=911ed82f4a3a4ad4938e4b1c8608e51b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:21.501029 26538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63c1903111784cf38265e2eeb4faff2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:21.500705 26669 tablet_service.cc:1467] Processing CreateTablet for tablet bbe54b53e86a48fbb1ba02074fb7afe3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=911ed82f4a3a4ad4938e4b1c8608e51b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:21.502234 26669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbe54b53e86a48fbb1ba02074fb7afe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:21.503432 26668 tablet_service.cc:1467] Processing CreateTablet for tablet 63c1903111784cf38265e2eeb4faff2e (DEFAULT_TABLE table=rebalance_test_table_0 [id=911ed82f4a3a4ad4938e4b1c8608e51b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:21.503969 26668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63c1903111784cf38265e2eeb4faff2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:21.521746 27416 tablet_bootstrap.cc:492] T 63c1903111784cf38265e2eeb4faff2e P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:21.526386 27415 tablet_bootstrap.cc:492] T 52061de1d76a420db5f3eb0aad70bf3a P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:21.527197 26405 tablet_service.cc:1467] Processing CreateTablet for tablet 52061de1d76a420db5f3eb0aad70bf3a (DEFAULT_TABLE table=rebalance_test_table_0 [id=911ed82f4a3a4ad4938e4b1c8608e51b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:21.528960 26407 tablet_service.cc:1467] Processing CreateTablet for tablet bbe54b53e86a48fbb1ba02074fb7afe3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=911ed82f4a3a4ad4938e4b1c8608e51b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:21.529143 27416 tablet_bootstrap.cc:654] T 63c1903111784cf38265e2eeb4faff2e P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:21.529475 27417 tablet_bootstrap.cc:492] T 63c1903111784cf38265e2eeb4faff2e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:21.530040 26405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52061de1d76a420db5f3eb0aad70bf3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:21.530732 27416 log.cc:824] T 63c1903111784cf38265e2eeb4faff2e P ba187e38a2754d60a12de9ac4856430a: Log is configured to *not* fsync() on all Append() calls
I20240502 14:14:21.531795 27415 tablet_bootstrap.cc:654] T 52061de1d76a420db5f3eb0aad70bf3a P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:21.532837 26407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbe54b53e86a48fbb1ba02074fb7afe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:21.533432 27417 tablet_bootstrap.cc:654] T 63c1903111784cf38265e2eeb4faff2e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:21.533413 27415 log.cc:824] T 52061de1d76a420db5f3eb0aad70bf3a P e4cfb8fc5a474177af6848a6efa13a27: Log is configured to *not* fsync() on all Append() calls
I20240502 14:14:21.533588 26406 tablet_service.cc:1467] Processing CreateTablet for tablet 63c1903111784cf38265e2eeb4faff2e (DEFAULT_TABLE table=rebalance_test_table_0 [id=911ed82f4a3a4ad4938e4b1c8608e51b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:21.534144 26406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63c1903111784cf38265e2eeb4faff2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:21.534907 27417 log.cc:824] T 63c1903111784cf38265e2eeb4faff2e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log is configured to *not* fsync() on all Append() calls
I20240502 14:14:21.541195 27416 tablet_bootstrap.cc:492] T 63c1903111784cf38265e2eeb4faff2e P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:21.541715 27416 ts_tablet_manager.cc:1367] T 63c1903111784cf38265e2eeb4faff2e P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.020s user 0.015s sys 0.000s
I20240502 14:14:21.544809 27417 tablet_bootstrap.cc:492] T 63c1903111784cf38265e2eeb4faff2e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:21.545231 27417 ts_tablet_manager.cc:1367] T 63c1903111784cf38265e2eeb4faff2e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.011s
I20240502 14:14:21.554451 27415 tablet_bootstrap.cc:492] T 52061de1d76a420db5f3eb0aad70bf3a P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:21.554806 27415 ts_tablet_manager.cc:1367] T 52061de1d76a420db5f3eb0aad70bf3a P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.029s user 0.010s sys 0.003s
I20240502 14:14:21.556427 27421 tablet_bootstrap.cc:492] T bbe54b53e86a48fbb1ba02074fb7afe3 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:21.556694 27417 raft_consensus.cc:348] T 63c1903111784cf38265e2eeb4faff2e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:21.557335 27417 raft_consensus.cc:374] T 63c1903111784cf38265e2eeb4faff2e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:21.557499 27417 raft_consensus.cc:729] T 63c1903111784cf38265e2eeb4faff2e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:21.558938 27417 consensus_queue.cc:260] T 63c1903111784cf38265e2eeb4faff2e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:21.557797 27416 raft_consensus.cc:348] T 63c1903111784cf38265e2eeb4faff2e P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:21.559505 27416 raft_consensus.cc:374] T 63c1903111784cf38265e2eeb4faff2e P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:21.559648 27416 raft_consensus.cc:729] T 63c1903111784cf38265e2eeb4faff2e P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:21.560494 27416 consensus_queue.cc:260] T 63c1903111784cf38265e2eeb4faff2e P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:21.561940 27421 tablet_bootstrap.cc:654] T bbe54b53e86a48fbb1ba02074fb7afe3 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:21.563701 27421 log.cc:824] T bbe54b53e86a48fbb1ba02074fb7afe3 P 89af7f423cf44717a08d6b142f0ed49d: Log is configured to *not* fsync() on all Append() calls
I20240502 14:14:21.565160 27415 raft_consensus.cc:348] T 52061de1d76a420db5f3eb0aad70bf3a P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:21.565826 27415 raft_consensus.cc:374] T 52061de1d76a420db5f3eb0aad70bf3a P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:21.566025 27415 raft_consensus.cc:729] T 52061de1d76a420db5f3eb0aad70bf3a P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:21.567128 27415 consensus_queue.cc:260] T 52061de1d76a420db5f3eb0aad70bf3a P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:21.567754 27417 ts_tablet_manager.cc:1398] T 63c1903111784cf38265e2eeb4faff2e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.022s user 0.009s sys 0.010s
I20240502 14:14:21.568275 27417 tablet_bootstrap.cc:492] T bbe54b53e86a48fbb1ba02074fb7afe3 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:21.570120 27417 tablet_bootstrap.cc:654] T bbe54b53e86a48fbb1ba02074fb7afe3 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:21.573829 27417 tablet_bootstrap.cc:492] T bbe54b53e86a48fbb1ba02074fb7afe3 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:21.574049 27417 ts_tablet_manager.cc:1367] T bbe54b53e86a48fbb1ba02074fb7afe3 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 14:14:21.574751 27416 ts_tablet_manager.cc:1398] T 63c1903111784cf38265e2eeb4faff2e P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.033s user 0.018s sys 0.004s
I20240502 14:14:21.575227 27417 raft_consensus.cc:348] T bbe54b53e86a48fbb1ba02074fb7afe3 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:21.575398 27416 tablet_bootstrap.cc:492] T 52061de1d76a420db5f3eb0aad70bf3a P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:21.575542 27417 raft_consensus.cc:374] T bbe54b53e86a48fbb1ba02074fb7afe3 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:21.575732 27417 raft_consensus.cc:729] T bbe54b53e86a48fbb1ba02074fb7afe3 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:21.576124 27417 consensus_queue.cc:260] T bbe54b53e86a48fbb1ba02074fb7afe3 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:21.576871 27417 ts_tablet_manager.cc:1398] T bbe54b53e86a48fbb1ba02074fb7afe3 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:21.580034 27421 tablet_bootstrap.cc:492] T bbe54b53e86a48fbb1ba02074fb7afe3 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:21.580335 27421 ts_tablet_manager.cc:1367] T bbe54b53e86a48fbb1ba02074fb7afe3 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.006s
I20240502 14:14:21.582336 27416 tablet_bootstrap.cc:654] T 52061de1d76a420db5f3eb0aad70bf3a P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:21.585006 27416 tablet_bootstrap.cc:492] T 52061de1d76a420db5f3eb0aad70bf3a P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:21.585242 27416 ts_tablet_manager.cc:1367] T 52061de1d76a420db5f3eb0aad70bf3a P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.001s
I20240502 14:14:21.585223 27415 ts_tablet_manager.cc:1398] T 52061de1d76a420db5f3eb0aad70bf3a P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.030s user 0.023s sys 0.009s
I20240502 14:14:21.585691 27415 tablet_bootstrap.cc:492] T bbe54b53e86a48fbb1ba02074fb7afe3 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:21.586782 27416 raft_consensus.cc:348] T 52061de1d76a420db5f3eb0aad70bf3a P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:21.587081 27416 raft_consensus.cc:374] T 52061de1d76a420db5f3eb0aad70bf3a P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:21.587416 27415 tablet_bootstrap.cc:654] T bbe54b53e86a48fbb1ba02074fb7afe3 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:21.587461 27416 raft_consensus.cc:729] T 52061de1d76a420db5f3eb0aad70bf3a P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:21.587852 27416 consensus_queue.cc:260] T 52061de1d76a420db5f3eb0aad70bf3a P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:21.588503 27421 raft_consensus.cc:348] T bbe54b53e86a48fbb1ba02074fb7afe3 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:21.589051 27421 raft_consensus.cc:374] T bbe54b53e86a48fbb1ba02074fb7afe3 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:21.589221 27421 raft_consensus.cc:729] T bbe54b53e86a48fbb1ba02074fb7afe3 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:21.590039 27416 ts_tablet_manager.cc:1398] T 52061de1d76a420db5f3eb0aad70bf3a P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 14:14:21.590137 27421 consensus_queue.cc:260] T bbe54b53e86a48fbb1ba02074fb7afe3 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:21.591516 27415 tablet_bootstrap.cc:492] T bbe54b53e86a48fbb1ba02074fb7afe3 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:21.591670 27415 ts_tablet_manager.cc:1367] T bbe54b53e86a48fbb1ba02074fb7afe3 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20240502 14:14:21.592628 27421 ts_tablet_manager.cc:1398] T bbe54b53e86a48fbb1ba02074fb7afe3 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20240502 14:14:21.592720 27415 raft_consensus.cc:348] T bbe54b53e86a48fbb1ba02074fb7afe3 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:21.592949 27415 raft_consensus.cc:374] T bbe54b53e86a48fbb1ba02074fb7afe3 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:21.593041 27415 raft_consensus.cc:729] T bbe54b53e86a48fbb1ba02074fb7afe3 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:21.593097 27421 tablet_bootstrap.cc:492] T 63c1903111784cf38265e2eeb4faff2e P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:21.593262 27415 consensus_queue.cc:260] T bbe54b53e86a48fbb1ba02074fb7afe3 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:21.593783 27415 ts_tablet_manager.cc:1398] T bbe54b53e86a48fbb1ba02074fb7afe3 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:21.594856 27421 tablet_bootstrap.cc:654] T 63c1903111784cf38265e2eeb4faff2e P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:21.597020 27421 tablet_bootstrap.cc:492] T 63c1903111784cf38265e2eeb4faff2e P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:21.597177 27421 ts_tablet_manager.cc:1367] T 63c1903111784cf38265e2eeb4faff2e P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:14:21.597904 27421 raft_consensus.cc:348] T 63c1903111784cf38265e2eeb4faff2e P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:21.598062 27421 raft_consensus.cc:374] T 63c1903111784cf38265e2eeb4faff2e P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:21.598156 27421 raft_consensus.cc:729] T 63c1903111784cf38265e2eeb4faff2e P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:21.598366 27421 consensus_queue.cc:260] T 63c1903111784cf38265e2eeb4faff2e P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:21.598824 27421 ts_tablet_manager.cc:1398] T 63c1903111784cf38265e2eeb4faff2e P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:14:21.599095 27421 tablet_bootstrap.cc:492] T 52061de1d76a420db5f3eb0aad70bf3a P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:21.600369 27421 tablet_bootstrap.cc:654] T 52061de1d76a420db5f3eb0aad70bf3a P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:21.602389 27421 tablet_bootstrap.cc:492] T 52061de1d76a420db5f3eb0aad70bf3a P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:21.602520 27421 ts_tablet_manager.cc:1367] T 52061de1d76a420db5f3eb0aad70bf3a P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20240502 14:14:21.603062 27421 raft_consensus.cc:348] T 52061de1d76a420db5f3eb0aad70bf3a P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:21.603276 27421 raft_consensus.cc:374] T 52061de1d76a420db5f3eb0aad70bf3a P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:21.603351 27421 raft_consensus.cc:729] T 52061de1d76a420db5f3eb0aad70bf3a P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:21.603583 27421 consensus_queue.cc:260] T 52061de1d76a420db5f3eb0aad70bf3a P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:21.604163 27421 ts_tablet_manager.cc:1398] T 52061de1d76a420db5f3eb0aad70bf3a P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20240502 14:14:21.693445 26473 tablet.cc:2369] T 52061de1d76a420db5f3eb0aad70bf3a P 89af7f423cf44717a08d6b142f0ed49d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 14:14:21.701485 26735 tablet.cc:2369] T 63c1903111784cf38265e2eeb4faff2e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:14:21.702261 27422 raft_consensus.cc:482] T 63c1903111784cf38265e2eeb4faff2e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:21.702462 27422 raft_consensus.cc:504] T 63c1903111784cf38265e2eeb4faff2e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:21.704066 27422 leader_election.cc:288] T 63c1903111784cf38265e2eeb4faff2e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
W20240502 14:14:21.708529 26604 tablet.cc:2369] T 52061de1d76a420db5f3eb0aad70bf3a P ba187e38a2754d60a12de9ac4856430a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:14:21.711470 26427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63c1903111784cf38265e2eeb4faff2e" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:14:21.711527 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63c1903111784cf38265e2eeb4faff2e" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:21.711934 26427 raft_consensus.cc:2454] T 63c1903111784cf38265e2eeb4faff2e P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:14:21.711951 26558 raft_consensus.cc:2454] T 63c1903111784cf38265e2eeb4faff2e P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:14:21.712585 26624 leader_election.cc:302] T 63c1903111784cf38265e2eeb4faff2e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:14:21.712954 27422 raft_consensus.cc:2781] T 63c1903111784cf38265e2eeb4faff2e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:21.713099 27422 raft_consensus.cc:482] T 63c1903111784cf38265e2eeb4faff2e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:21.713208 27422 raft_consensus.cc:3037] T 63c1903111784cf38265e2eeb4faff2e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:21.715211 27422 raft_consensus.cc:504] T 63c1903111784cf38265e2eeb4faff2e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:21.715763 27422 leader_election.cc:288] T 63c1903111784cf38265e2eeb4faff2e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Requested vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:14:21.716207 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63c1903111784cf38265e2eeb4faff2e" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:14:21.716298 26427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63c1903111784cf38265e2eeb4faff2e" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:14:21.716452 26558 raft_consensus.cc:3037] T 63c1903111784cf38265e2eeb4faff2e P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:21.716611 26427 raft_consensus.cc:3037] T 63c1903111784cf38265e2eeb4faff2e P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:21.718961 26558 raft_consensus.cc:2454] T 63c1903111784cf38265e2eeb4faff2e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:14:21.719071 26427 raft_consensus.cc:2454] T 63c1903111784cf38265e2eeb4faff2e P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:14:21.719353 26624 leader_election.cc:302] T 63c1903111784cf38265e2eeb4faff2e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:21.719650 27422 raft_consensus.cc:2781] T 63c1903111784cf38265e2eeb4faff2e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:21.720103 27422 raft_consensus.cc:686] T 63c1903111784cf38265e2eeb4faff2e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Becoming Leader. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Running, Role: LEADER
I20240502 14:14:21.720607 27422 consensus_queue.cc:237] T 63c1903111784cf38265e2eeb4faff2e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
W20240502 14:14:21.721745 26865 tablet.cc:2369] T 52061de1d76a420db5f3eb0aad70bf3a P e4cfb8fc5a474177af6848a6efa13a27: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:14:21.725545 26284 catalog_manager.cc:5484] T 63c1903111784cf38265e2eeb4faff2e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195). New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:14:21.776660 27429 raft_consensus.cc:482] T bbe54b53e86a48fbb1ba02074fb7afe3 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:21.776866 27429 raft_consensus.cc:504] T bbe54b53e86a48fbb1ba02074fb7afe3 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:21.778555 27429 leader_election.cc:288] T bbe54b53e86a48fbb1ba02074fb7afe3 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:14:21.787156 26819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe54b53e86a48fbb1ba02074fb7afe3" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:14:21.787127 26689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe54b53e86a48fbb1ba02074fb7afe3" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:14:21.787556 26689 raft_consensus.cc:2454] T bbe54b53e86a48fbb1ba02074fb7afe3 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:14:21.787644 26819 raft_consensus.cc:2454] T bbe54b53e86a48fbb1ba02074fb7afe3 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:14:21.788133 26363 leader_election.cc:302] T bbe54b53e86a48fbb1ba02074fb7afe3 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:14:21.788591 27429 raft_consensus.cc:2781] T bbe54b53e86a48fbb1ba02074fb7afe3 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:21.788707 27429 raft_consensus.cc:482] T bbe54b53e86a48fbb1ba02074fb7afe3 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:21.788802 27429 raft_consensus.cc:3037] T bbe54b53e86a48fbb1ba02074fb7afe3 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:21.790839 27429 raft_consensus.cc:504] T bbe54b53e86a48fbb1ba02074fb7afe3 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:21.791419 27429 leader_election.cc:288] T bbe54b53e86a48fbb1ba02074fb7afe3 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Requested vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:14:21.791895 26819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe54b53e86a48fbb1ba02074fb7afe3" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:14:21.791978 26689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe54b53e86a48fbb1ba02074fb7afe3" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:14:21.792065 26819 raft_consensus.cc:3037] T bbe54b53e86a48fbb1ba02074fb7afe3 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:21.792187 26689 raft_consensus.cc:3037] T bbe54b53e86a48fbb1ba02074fb7afe3 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:21.794229 26819 raft_consensus.cc:2454] T bbe54b53e86a48fbb1ba02074fb7afe3 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:14:21.794312 26689 raft_consensus.cc:2454] T bbe54b53e86a48fbb1ba02074fb7afe3 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:14:21.794661 26363 leader_election.cc:302] T bbe54b53e86a48fbb1ba02074fb7afe3 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:21.794989 27429 raft_consensus.cc:2781] T bbe54b53e86a48fbb1ba02074fb7afe3 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:21.795435 27429 raft_consensus.cc:686] T bbe54b53e86a48fbb1ba02074fb7afe3 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Becoming Leader. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Running, Role: LEADER
I20240502 14:14:21.795961 27429 consensus_queue.cc:237] T bbe54b53e86a48fbb1ba02074fb7afe3 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:21.800446 26284 catalog_manager.cc:5484] T bbe54b53e86a48fbb1ba02074fb7afe3 P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: term changed from 0 to 1, leader changed from <none> to 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193). New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:14:21.896476 27425 raft_consensus.cc:482] T 52061de1d76a420db5f3eb0aad70bf3a P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:21.896694 27425 raft_consensus.cc:504] T 52061de1d76a420db5f3eb0aad70bf3a P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:21.898418 27425 leader_election.cc:288] T 52061de1d76a420db5f3eb0aad70bf3a P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:14:21.905612 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52061de1d76a420db5f3eb0aad70bf3a" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:21.905819 26427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52061de1d76a420db5f3eb0aad70bf3a" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:14:21.905905 26558 raft_consensus.cc:2454] T 52061de1d76a420db5f3eb0aad70bf3a P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 0.
I20240502 14:14:21.906103 26427 raft_consensus.cc:2454] T 52061de1d76a420db5f3eb0aad70bf3a P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 0.
I20240502 14:14:21.906507 26754 leader_election.cc:302] T 52061de1d76a420db5f3eb0aad70bf3a P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:21.906932 27425 raft_consensus.cc:2781] T 52061de1d76a420db5f3eb0aad70bf3a P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:21.907081 27425 raft_consensus.cc:482] T 52061de1d76a420db5f3eb0aad70bf3a P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:21.907199 27425 raft_consensus.cc:3037] T 52061de1d76a420db5f3eb0aad70bf3a P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:21.909359 27425 raft_consensus.cc:504] T 52061de1d76a420db5f3eb0aad70bf3a P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:21.910041 27425 leader_election.cc:288] T 52061de1d76a420db5f3eb0aad70bf3a P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Requested vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:14:21.910570 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52061de1d76a420db5f3eb0aad70bf3a" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:14:21.910641 26427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52061de1d76a420db5f3eb0aad70bf3a" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:14:21.910807 26558 raft_consensus.cc:3037] T 52061de1d76a420db5f3eb0aad70bf3a P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:21.910861 26427 raft_consensus.cc:3037] T 52061de1d76a420db5f3eb0aad70bf3a P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:21.912767 26427 raft_consensus.cc:2454] T 52061de1d76a420db5f3eb0aad70bf3a P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
I20240502 14:14:21.912777 26558 raft_consensus.cc:2454] T 52061de1d76a420db5f3eb0aad70bf3a P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
I20240502 14:14:21.913215 26754 leader_election.cc:302] T 52061de1d76a420db5f3eb0aad70bf3a P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:21.913522 27425 raft_consensus.cc:2781] T 52061de1d76a420db5f3eb0aad70bf3a P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:21.913954 27425 raft_consensus.cc:686] T 52061de1d76a420db5f3eb0aad70bf3a P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Becoming Leader. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Running, Role: LEADER
I20240502 14:14:21.914423 27425 consensus_queue.cc:237] T 52061de1d76a420db5f3eb0aad70bf3a P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:21.918671 26284 catalog_manager.cc:5484] T 52061de1d76a420db5f3eb0aad70bf3a P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196). New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:14:22.199553 27422 consensus_queue.cc:1035] T 63c1903111784cf38265e2eeb4faff2e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:22.204020 27429 consensus_queue.cc:1035] T bbe54b53e86a48fbb1ba02074fb7afe3 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:22.238756 27438 consensus_queue.cc:1035] T bbe54b53e86a48fbb1ba02074fb7afe3 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:22.259918 27450 consensus_queue.cc:1035] T 63c1903111784cf38265e2eeb4faff2e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:22.318382 27444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:22.318809 27444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:22.327119 27444 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:14:22.362423 27425 consensus_queue.cc:1035] T 52061de1d76a420db5f3eb0aad70bf3a P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:22.368705 27425 consensus_queue.cc:1035] T 52061de1d76a420db5f3eb0aad70bf3a P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 248.098 ms
time per row: 0.124049 ms
I20240502 14:14:22.750311 26280 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:22.759621 26538 tablet_service.cc:1467] Processing CreateTablet for tablet 946239523c0646f1835d7a85b4e7d117 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d7c1bbb69be74fd3a8e809dd8649698d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:22.760139 26538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 946239523c0646f1835d7a85b4e7d117. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:22.760114 26668 tablet_service.cc:1467] Processing CreateTablet for tablet 946239523c0646f1835d7a85b4e7d117 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d7c1bbb69be74fd3a8e809dd8649698d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:22.760547 26668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 946239523c0646f1835d7a85b4e7d117. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:22.761027 26406 tablet_service.cc:1467] Processing CreateTablet for tablet 0fc2e6c74dc34a45ae1ec46608642880 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d7c1bbb69be74fd3a8e809dd8649698d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:22.761152 26798 tablet_service.cc:1467] Processing CreateTablet for tablet 946239523c0646f1835d7a85b4e7d117 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d7c1bbb69be74fd3a8e809dd8649698d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:22.761469 26406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fc2e6c74dc34a45ae1ec46608642880. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:22.761512 26798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 946239523c0646f1835d7a85b4e7d117. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:22.764853 26537 tablet_service.cc:1467] Processing CreateTablet for tablet 65e69adddb7c4dc59e62f616e880cdbc (DEFAULT_TABLE table=rebalance_test_table_1 [id=d7c1bbb69be74fd3a8e809dd8649698d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:22.765300 26537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65e69adddb7c4dc59e62f616e880cdbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:22.765766 26669 tablet_service.cc:1467] Processing CreateTablet for tablet 65e69adddb7c4dc59e62f616e880cdbc (DEFAULT_TABLE table=rebalance_test_table_1 [id=d7c1bbb69be74fd3a8e809dd8649698d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:22.765837 26536 tablet_service.cc:1467] Processing CreateTablet for tablet 0fc2e6c74dc34a45ae1ec46608642880 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d7c1bbb69be74fd3a8e809dd8649698d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:22.765893 26799 tablet_service.cc:1467] Processing CreateTablet for tablet 65e69adddb7c4dc59e62f616e880cdbc (DEFAULT_TABLE table=rebalance_test_table_1 [id=d7c1bbb69be74fd3a8e809dd8649698d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:22.766129 26669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65e69adddb7c4dc59e62f616e880cdbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:22.766283 26799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65e69adddb7c4dc59e62f616e880cdbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:22.766351 26536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fc2e6c74dc34a45ae1ec46608642880. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:22.777032 27512 tablet_bootstrap.cc:492] T 65e69adddb7c4dc59e62f616e880cdbc P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:22.779423 27512 tablet_bootstrap.cc:654] T 65e69adddb7c4dc59e62f616e880cdbc P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:22.781293 27513 tablet_bootstrap.cc:492] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:22.781339 26669 tablet_service.cc:1467] Processing CreateTablet for tablet 0fc2e6c74dc34a45ae1ec46608642880 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d7c1bbb69be74fd3a8e809dd8649698d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:22.781945 26669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fc2e6c74dc34a45ae1ec46608642880. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:22.782673 27511 tablet_bootstrap.cc:492] T 65e69adddb7c4dc59e62f616e880cdbc P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:22.783635 27513 tablet_bootstrap.cc:654] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:22.785596 27511 tablet_bootstrap.cc:654] T 65e69adddb7c4dc59e62f616e880cdbc P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:22.787339 27514 tablet_bootstrap.cc:492] T 946239523c0646f1835d7a85b4e7d117 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:22.789821 27514 tablet_bootstrap.cc:654] T 946239523c0646f1835d7a85b4e7d117 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:22.791923 27511 tablet_bootstrap.cc:492] T 65e69adddb7c4dc59e62f616e880cdbc P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:22.792088 27511 ts_tablet_manager.cc:1367] T 65e69adddb7c4dc59e62f616e880cdbc P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20240502 14:14:22.793357 27511 raft_consensus.cc:348] T 65e69adddb7c4dc59e62f616e880cdbc P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:22.793540 27511 raft_consensus.cc:374] T 65e69adddb7c4dc59e62f616e880cdbc P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:22.793649 27511 raft_consensus.cc:729] T 65e69adddb7c4dc59e62f616e880cdbc P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:22.793898 27513 tablet_bootstrap.cc:492] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:22.794064 27513 ts_tablet_manager.cc:1367] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.008s
I20240502 14:14:22.793942 27511 consensus_queue.cc:260] T 65e69adddb7c4dc59e62f616e880cdbc P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:22.794493 27512 tablet_bootstrap.cc:492] T 65e69adddb7c4dc59e62f616e880cdbc P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:22.794639 27512 ts_tablet_manager.cc:1367] T 65e69adddb7c4dc59e62f616e880cdbc P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.003s
I20240502 14:14:22.794732 27511 ts_tablet_manager.cc:1398] T 65e69adddb7c4dc59e62f616e880cdbc P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:22.794857 27513 raft_consensus.cc:348] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:22.795025 27511 tablet_bootstrap.cc:492] T 0fc2e6c74dc34a45ae1ec46608642880 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:22.795071 27513 raft_consensus.cc:374] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:22.795167 27513 raft_consensus.cc:729] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:22.795684 27513 consensus_queue.cc:260] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:22.795403 27512 raft_consensus.cc:348] T 65e69adddb7c4dc59e62f616e880cdbc P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:22.796231 27512 raft_consensus.cc:374] T 65e69adddb7c4dc59e62f616e880cdbc P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:22.796350 27512 raft_consensus.cc:729] T 65e69adddb7c4dc59e62f616e880cdbc P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:22.796645 27511 tablet_bootstrap.cc:654] T 0fc2e6c74dc34a45ae1ec46608642880 P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:22.796679 27512 consensus_queue.cc:260] T 65e69adddb7c4dc59e62f616e880cdbc P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:22.797787 27513 ts_tablet_manager.cc:1398] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 14:14:22.798450 27512 ts_tablet_manager.cc:1398] T 65e69adddb7c4dc59e62f616e880cdbc P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:14:22.798818 27512 tablet_bootstrap.cc:492] T 946239523c0646f1835d7a85b4e7d117 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:22.797164 27514 tablet_bootstrap.cc:492] T 946239523c0646f1835d7a85b4e7d117 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:22.799216 27514 ts_tablet_manager.cc:1367] T 946239523c0646f1835d7a85b4e7d117 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20240502 14:14:22.800058 27514 raft_consensus.cc:348] T 946239523c0646f1835d7a85b4e7d117 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:22.800513 27514 raft_consensus.cc:374] T 946239523c0646f1835d7a85b4e7d117 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:22.800681 27514 raft_consensus.cc:729] T 946239523c0646f1835d7a85b4e7d117 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:22.800982 27512 tablet_bootstrap.cc:654] T 946239523c0646f1835d7a85b4e7d117 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:22.800988 27514 consensus_queue.cc:260] T 946239523c0646f1835d7a85b4e7d117 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:22.801678 27514 ts_tablet_manager.cc:1398] T 946239523c0646f1835d7a85b4e7d117 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:14:22.802065 27514 tablet_bootstrap.cc:492] T 65e69adddb7c4dc59e62f616e880cdbc P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:22.803833 27514 tablet_bootstrap.cc:654] T 65e69adddb7c4dc59e62f616e880cdbc P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:22.807490 27512 tablet_bootstrap.cc:492] T 946239523c0646f1835d7a85b4e7d117 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:22.807721 27512 ts_tablet_manager.cc:1367] T 946239523c0646f1835d7a85b4e7d117 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20240502 14:14:22.808344 27514 tablet_bootstrap.cc:492] T 65e69adddb7c4dc59e62f616e880cdbc P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:22.808349 27511 tablet_bootstrap.cc:492] T 0fc2e6c74dc34a45ae1ec46608642880 P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:22.808313 27512 raft_consensus.cc:348] T 946239523c0646f1835d7a85b4e7d117 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:22.808538 27514 ts_tablet_manager.cc:1367] T 65e69adddb7c4dc59e62f616e880cdbc P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20240502 14:14:22.808544 27511 ts_tablet_manager.cc:1367] T 0fc2e6c74dc34a45ae1ec46608642880 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.009s
I20240502 14:14:22.808588 27512 raft_consensus.cc:374] T 946239523c0646f1835d7a85b4e7d117 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:22.808689 27512 raft_consensus.cc:729] T 946239523c0646f1835d7a85b4e7d117 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:22.808943 27512 consensus_queue.cc:260] T 946239523c0646f1835d7a85b4e7d117 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:22.809412 27512 ts_tablet_manager.cc:1398] T 946239523c0646f1835d7a85b4e7d117 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:22.809296 27514 raft_consensus.cc:348] T 65e69adddb7c4dc59e62f616e880cdbc P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:22.809309 27511 raft_consensus.cc:348] T 0fc2e6c74dc34a45ae1ec46608642880 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:22.809511 27511 raft_consensus.cc:374] T 0fc2e6c74dc34a45ae1ec46608642880 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:22.809511 27514 raft_consensus.cc:374] T 65e69adddb7c4dc59e62f616e880cdbc P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:22.809597 27514 raft_consensus.cc:729] T 65e69adddb7c4dc59e62f616e880cdbc P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:22.809607 27511 raft_consensus.cc:729] T 0fc2e6c74dc34a45ae1ec46608642880 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:22.809691 27512 tablet_bootstrap.cc:492] T 0fc2e6c74dc34a45ae1ec46608642880 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:22.809860 27511 consensus_queue.cc:260] T 0fc2e6c74dc34a45ae1ec46608642880 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:22.809875 27514 consensus_queue.cc:260] T 65e69adddb7c4dc59e62f616e880cdbc P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:22.810388 27511 ts_tablet_manager.cc:1398] T 0fc2e6c74dc34a45ae1ec46608642880 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:14:22.811159 27512 tablet_bootstrap.cc:654] T 0fc2e6c74dc34a45ae1ec46608642880 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:22.811295 27511 tablet_bootstrap.cc:492] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:22.812974 27511 tablet_bootstrap.cc:654] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:22.813159 27514 ts_tablet_manager.cc:1398] T 65e69adddb7c4dc59e62f616e880cdbc P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 14:14:22.815809 27512 tablet_bootstrap.cc:492] T 0fc2e6c74dc34a45ae1ec46608642880 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:22.816012 27512 ts_tablet_manager.cc:1367] T 0fc2e6c74dc34a45ae1ec46608642880 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.004s
I20240502 14:14:22.816334 27511 tablet_bootstrap.cc:492] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:22.816499 27511 ts_tablet_manager.cc:1367] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240502 14:14:22.816864 27512 raft_consensus.cc:348] T 0fc2e6c74dc34a45ae1ec46608642880 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:22.817018 27512 raft_consensus.cc:374] T 0fc2e6c74dc34a45ae1ec46608642880 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:22.817109 27512 raft_consensus.cc:729] T 0fc2e6c74dc34a45ae1ec46608642880 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:22.817140 27511 raft_consensus.cc:348] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:22.817344 27511 raft_consensus.cc:374] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:22.817312 27512 consensus_queue.cc:260] T 0fc2e6c74dc34a45ae1ec46608642880 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:22.817443 27511 raft_consensus.cc:729] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:22.817690 27511 consensus_queue.cc:260] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:22.817829 27512 ts_tablet_manager.cc:1398] T 0fc2e6c74dc34a45ae1ec46608642880 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:14:22.818269 27511 ts_tablet_manager.cc:1398] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:22.863502 27445 raft_consensus.cc:482] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:22.863750 27445 raft_consensus.cc:504] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:22.865485 27445 leader_election.cc:288] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:14:22.870995 27438 raft_consensus.cc:482] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:22.871213 27438 raft_consensus.cc:504] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:22.872094 27438 leader_election.cc:288] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:14:22.873047 26689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc2e6c74dc34a45ae1ec46608642880" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:14:22.873369 26689 raft_consensus.cc:2454] T 0fc2e6c74dc34a45ae1ec46608642880 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:14:22.873957 26363 leader_election.cc:302] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:14:22.874368 27438 raft_consensus.cc:2781] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:22.874472 27438 raft_consensus.cc:482] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:22.874590 27438 raft_consensus.cc:3037] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:22.875612 26819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "946239523c0646f1835d7a85b4e7d117" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:14:22.875890 26819 raft_consensus.cc:2454] T 946239523c0646f1835d7a85b4e7d117 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 0.
I20240502 14:14:22.876662 26689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "946239523c0646f1835d7a85b4e7d117" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:14:22.876907 26689 raft_consensus.cc:2454] T 946239523c0646f1835d7a85b4e7d117 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 0.
I20240502 14:14:22.876834 26494 leader_election.cc:302] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:22.877344 27445 raft_consensus.cc:2781] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:22.877478 27445 raft_consensus.cc:482] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:22.877600 27445 raft_consensus.cc:3037] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:22.877748 27438 raft_consensus.cc:504] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:22.878669 27438 leader_election.cc:288] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Requested vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:14:22.878978 26689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc2e6c74dc34a45ae1ec46608642880" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:14:22.879205 26689 raft_consensus.cc:3037] T 0fc2e6c74dc34a45ae1ec46608642880 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:22.880786 27445 raft_consensus.cc:504] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:22.881244 26689 raft_consensus.cc:2454] T 0fc2e6c74dc34a45ae1ec46608642880 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:14:22.881533 27445 leader_election.cc:288] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Requested vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:14:22.881709 26363 leader_election.cc:302] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:14:22.882040 27438 raft_consensus.cc:2781] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:22.882090 26819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "946239523c0646f1835d7a85b4e7d117" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:14:22.882228 27438 raft_consensus.cc:686] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Becoming Leader. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Running, Role: LEADER
I20240502 14:14:22.882323 26819 raft_consensus.cc:3037] T 946239523c0646f1835d7a85b4e7d117 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:22.882508 27438 consensus_queue.cc:237] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:22.883116 26688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "946239523c0646f1835d7a85b4e7d117" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:14:22.883414 26688 raft_consensus.cc:3037] T 946239523c0646f1835d7a85b4e7d117 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:22.883790 26557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc2e6c74dc34a45ae1ec46608642880" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:22.883889 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc2e6c74dc34a45ae1ec46608642880" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:14:22.884034 26557 raft_consensus.cc:2454] T 0fc2e6c74dc34a45ae1ec46608642880 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:14:22.885345 26819 raft_consensus.cc:2454] T 946239523c0646f1835d7a85b4e7d117 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:14:22.885911 26494 leader_election.cc:302] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:22.886266 27445 raft_consensus.cc:2781] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:22.886420 26688 raft_consensus.cc:2454] T 946239523c0646f1835d7a85b4e7d117 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:14:22.887413 26281 catalog_manager.cc:5484] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: term changed from 0 to 1, leader changed from <none> to 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193). New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:14:22.888005 27445 raft_consensus.cc:686] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Becoming Leader. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Running, Role: LEADER
I20240502 14:14:22.888592 27445 consensus_queue.cc:237] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:22.891973 26281 catalog_manager.cc:5484] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a reported cstate change: term changed from 0 to 1, leader changed from <none> to ba187e38a2754d60a12de9ac4856430a (127.19.94.194). New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:14:22.909814 27445 raft_consensus.cc:482] T 65e69adddb7c4dc59e62f616e880cdbc P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:22.910039 27445 raft_consensus.cc:504] T 65e69adddb7c4dc59e62f616e880cdbc P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:22.910636 27445 leader_election.cc:288] T 65e69adddb7c4dc59e62f616e880cdbc P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:14:22.911327 26819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65e69adddb7c4dc59e62f616e880cdbc" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:14:22.911378 26688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65e69adddb7c4dc59e62f616e880cdbc" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:14:22.911545 26819 raft_consensus.cc:2454] T 65e69adddb7c4dc59e62f616e880cdbc P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 0.
I20240502 14:14:22.911648 26688 raft_consensus.cc:2454] T 65e69adddb7c4dc59e62f616e880cdbc P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 0.
I20240502 14:14:22.912006 26494 leader_election.cc:302] T 65e69adddb7c4dc59e62f616e880cdbc P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:22.912341 27445 raft_consensus.cc:2781] T 65e69adddb7c4dc59e62f616e880cdbc P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:22.912475 27445 raft_consensus.cc:482] T 65e69adddb7c4dc59e62f616e880cdbc P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:22.912588 27445 raft_consensus.cc:3037] T 65e69adddb7c4dc59e62f616e880cdbc P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:22.914475 27445 raft_consensus.cc:504] T 65e69adddb7c4dc59e62f616e880cdbc P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:22.915107 27445 leader_election.cc:288] T 65e69adddb7c4dc59e62f616e880cdbc P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Requested vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:14:22.915572 26819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65e69adddb7c4dc59e62f616e880cdbc" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:14:22.915630 26688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65e69adddb7c4dc59e62f616e880cdbc" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:14:22.915781 26819 raft_consensus.cc:3037] T 65e69adddb7c4dc59e62f616e880cdbc P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:22.915828 26688 raft_consensus.cc:3037] T 65e69adddb7c4dc59e62f616e880cdbc P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:22.917732 26688 raft_consensus.cc:2454] T 65e69adddb7c4dc59e62f616e880cdbc P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:14:22.917745 26819 raft_consensus.cc:2454] T 65e69adddb7c4dc59e62f616e880cdbc P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:14:22.918144 26494 leader_election.cc:302] T 65e69adddb7c4dc59e62f616e880cdbc P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:22.918462 27445 raft_consensus.cc:2781] T 65e69adddb7c4dc59e62f616e880cdbc P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:22.918642 27445 raft_consensus.cc:686] T 65e69adddb7c4dc59e62f616e880cdbc P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Becoming Leader. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Running, Role: LEADER
I20240502 14:14:22.918905 27445 consensus_queue.cc:237] T 65e69adddb7c4dc59e62f616e880cdbc P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:22.921862 26281 catalog_manager.cc:5484] T 65e69adddb7c4dc59e62f616e880cdbc P ba187e38a2754d60a12de9ac4856430a reported cstate change: term changed from 0 to 1, leader changed from <none> to ba187e38a2754d60a12de9ac4856430a (127.19.94.194). New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:14:23.276695 27445 consensus_queue.cc:1035] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:23.283094 27529 consensus_queue.cc:1035] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:23.291329 27532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:23.291792 27532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:23.299408 27532 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:14:23.320760 26557 raft_consensus.cc:3037] T 0fc2e6c74dc34a45ae1ec46608642880 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:23.323767 27438 consensus_queue.cc:1035] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:23.330299 27438 consensus_queue.cc:1035] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
I20240502 14:14:23.405225 27529 consensus_queue.cc:1035] T 65e69adddb7c4dc59e62f616e880cdbc P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:23.428861 27445 consensus_queue.cc:1035] T 65e69adddb7c4dc59e62f616e880cdbc P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 218.981 ms
time per row: 0.109491 ms
I20240502 14:14:23.676972 26281 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:23.686830 26798 tablet_service.cc:1467] Processing CreateTablet for tablet 52a4b0ee48204fd18ebb932448bd6503 (DEFAULT_TABLE table=rebalance_test_table_2 [id=003c373a4c124a06aae79946d6d5664c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:23.686841 26406 tablet_service.cc:1467] Processing CreateTablet for tablet 52a4b0ee48204fd18ebb932448bd6503 (DEFAULT_TABLE table=rebalance_test_table_2 [id=003c373a4c124a06aae79946d6d5664c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:23.687332 26406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52a4b0ee48204fd18ebb932448bd6503. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:23.687343 26798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52a4b0ee48204fd18ebb932448bd6503. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:23.687901 26537 tablet_service.cc:1467] Processing CreateTablet for tablet 52a4b0ee48204fd18ebb932448bd6503 (DEFAULT_TABLE table=rebalance_test_table_2 [id=003c373a4c124a06aae79946d6d5664c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:23.688314 26537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52a4b0ee48204fd18ebb932448bd6503. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:23.688597 26799 tablet_service.cc:1467] Processing CreateTablet for tablet 18c4a357e8814da8aac5b526bc920dab (DEFAULT_TABLE table=rebalance_test_table_2 [id=003c373a4c124a06aae79946d6d5664c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:23.689011 26799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18c4a357e8814da8aac5b526bc920dab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:23.690759 26797 tablet_service.cc:1467] Processing CreateTablet for tablet dcc8fde0e9ab4236a33971bbc90716ff (DEFAULT_TABLE table=rebalance_test_table_2 [id=003c373a4c124a06aae79946d6d5664c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:23.691185 26797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc8fde0e9ab4236a33971bbc90716ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:23.693099 26669 tablet_service.cc:1467] Processing CreateTablet for tablet 18c4a357e8814da8aac5b526bc920dab (DEFAULT_TABLE table=rebalance_test_table_2 [id=003c373a4c124a06aae79946d6d5664c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:23.693354 26536 tablet_service.cc:1467] Processing CreateTablet for tablet dcc8fde0e9ab4236a33971bbc90716ff (DEFAULT_TABLE table=rebalance_test_table_2 [id=003c373a4c124a06aae79946d6d5664c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:23.693536 26669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18c4a357e8814da8aac5b526bc920dab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:23.693742 26536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc8fde0e9ab4236a33971bbc90716ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:23.697293 26405 tablet_service.cc:1467] Processing CreateTablet for tablet 18c4a357e8814da8aac5b526bc920dab (DEFAULT_TABLE table=rebalance_test_table_2 [id=003c373a4c124a06aae79946d6d5664c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:23.697707 26405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18c4a357e8814da8aac5b526bc920dab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:23.700613 26407 tablet_service.cc:1467] Processing CreateTablet for tablet dcc8fde0e9ab4236a33971bbc90716ff (DEFAULT_TABLE table=rebalance_test_table_2 [id=003c373a4c124a06aae79946d6d5664c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:23.701018 26407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc8fde0e9ab4236a33971bbc90716ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:23.707062 27598 tablet_bootstrap.cc:492] T 52a4b0ee48204fd18ebb932448bd6503 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:23.709770 27598 tablet_bootstrap.cc:654] T 52a4b0ee48204fd18ebb932448bd6503 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:23.724898 27601 tablet_bootstrap.cc:492] T dcc8fde0e9ab4236a33971bbc90716ff P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:23.726322 27598 tablet_bootstrap.cc:492] T 52a4b0ee48204fd18ebb932448bd6503 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:23.726500 27598 ts_tablet_manager.cc:1367] T 52a4b0ee48204fd18ebb932448bd6503 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.000s
I20240502 14:14:23.726967 27601 tablet_bootstrap.cc:654] T dcc8fde0e9ab4236a33971bbc90716ff P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:23.727216 27598 raft_consensus.cc:348] T 52a4b0ee48204fd18ebb932448bd6503 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:23.727401 27598 raft_consensus.cc:374] T 52a4b0ee48204fd18ebb932448bd6503 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:23.727509 27598 raft_consensus.cc:729] T 52a4b0ee48204fd18ebb932448bd6503 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:23.727751 27598 consensus_queue.cc:260] T 52a4b0ee48204fd18ebb932448bd6503 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:23.729053 27601 tablet_bootstrap.cc:492] T dcc8fde0e9ab4236a33971bbc90716ff P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:23.729113 27598 ts_tablet_manager.cc:1398] T 52a4b0ee48204fd18ebb932448bd6503 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:23.729252 27601 ts_tablet_manager.cc:1367] T dcc8fde0e9ab4236a33971bbc90716ff P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240502 14:14:23.729452 27598 tablet_bootstrap.cc:492] T 18c4a357e8814da8aac5b526bc920dab P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:23.730105 27601 raft_consensus.cc:348] T dcc8fde0e9ab4236a33971bbc90716ff P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:23.730479 27601 raft_consensus.cc:374] T dcc8fde0e9ab4236a33971bbc90716ff P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:23.730626 27601 raft_consensus.cc:729] T dcc8fde0e9ab4236a33971bbc90716ff P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:23.730882 27599 tablet_bootstrap.cc:492] T dcc8fde0e9ab4236a33971bbc90716ff P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:23.730901 27601 consensus_queue.cc:260] T dcc8fde0e9ab4236a33971bbc90716ff P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:23.731624 27601 ts_tablet_manager.cc:1398] T dcc8fde0e9ab4236a33971bbc90716ff P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:23.731961 27601 tablet_bootstrap.cc:492] T 52a4b0ee48204fd18ebb932448bd6503 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:23.732777 27598 tablet_bootstrap.cc:654] T 18c4a357e8814da8aac5b526bc920dab P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:23.733254 27599 tablet_bootstrap.cc:654] T dcc8fde0e9ab4236a33971bbc90716ff P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:23.733628 27601 tablet_bootstrap.cc:654] T 52a4b0ee48204fd18ebb932448bd6503 P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:23.742513 27599 tablet_bootstrap.cc:492] T dcc8fde0e9ab4236a33971bbc90716ff P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:23.742779 27599 ts_tablet_manager.cc:1367] T dcc8fde0e9ab4236a33971bbc90716ff P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240502 14:14:23.743588 27599 raft_consensus.cc:348] T dcc8fde0e9ab4236a33971bbc90716ff P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:23.743820 27599 raft_consensus.cc:374] T dcc8fde0e9ab4236a33971bbc90716ff P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:23.743909 27599 raft_consensus.cc:729] T dcc8fde0e9ab4236a33971bbc90716ff P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:23.744141 27599 consensus_queue.cc:260] T dcc8fde0e9ab4236a33971bbc90716ff P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:23.745468 27601 tablet_bootstrap.cc:492] T 52a4b0ee48204fd18ebb932448bd6503 P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:23.745733 27601 ts_tablet_manager.cc:1367] T 52a4b0ee48204fd18ebb932448bd6503 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20240502 14:14:23.746179 27599 ts_tablet_manager.cc:1398] T dcc8fde0e9ab4236a33971bbc90716ff P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:23.746510 27599 tablet_bootstrap.cc:492] T 52a4b0ee48204fd18ebb932448bd6503 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:23.746631 27601 raft_consensus.cc:348] T 52a4b0ee48204fd18ebb932448bd6503 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:23.746910 27601 raft_consensus.cc:374] T 52a4b0ee48204fd18ebb932448bd6503 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:23.747014 27601 raft_consensus.cc:729] T 52a4b0ee48204fd18ebb932448bd6503 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:23.747324 27601 consensus_queue.cc:260] T 52a4b0ee48204fd18ebb932448bd6503 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:23.747977 27601 ts_tablet_manager.cc:1398] T 52a4b0ee48204fd18ebb932448bd6503 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:23.748059 27599 tablet_bootstrap.cc:654] T 52a4b0ee48204fd18ebb932448bd6503 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:23.749125 27597 tablet_bootstrap.cc:492] T 18c4a357e8814da8aac5b526bc920dab P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:23.749219 27598 tablet_bootstrap.cc:492] T 18c4a357e8814da8aac5b526bc920dab P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:23.749460 27598 ts_tablet_manager.cc:1367] T 18c4a357e8814da8aac5b526bc920dab P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.005s
I20240502 14:14:23.750332 27598 raft_consensus.cc:348] T 18c4a357e8814da8aac5b526bc920dab P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:23.750597 27598 raft_consensus.cc:374] T 18c4a357e8814da8aac5b526bc920dab P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:23.750725 27598 raft_consensus.cc:729] T 18c4a357e8814da8aac5b526bc920dab P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:23.750795 27597 tablet_bootstrap.cc:654] T 18c4a357e8814da8aac5b526bc920dab P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:23.751032 27598 consensus_queue.cc:260] T 18c4a357e8814da8aac5b526bc920dab P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:23.751791 27598 ts_tablet_manager.cc:1398] T 18c4a357e8814da8aac5b526bc920dab P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:23.752140 27598 tablet_bootstrap.cc:492] T dcc8fde0e9ab4236a33971bbc90716ff P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:23.753276 27599 tablet_bootstrap.cc:492] T 52a4b0ee48204fd18ebb932448bd6503 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:23.753489 27599 ts_tablet_manager.cc:1367] T 52a4b0ee48204fd18ebb932448bd6503 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240502 14:14:23.753779 27598 tablet_bootstrap.cc:654] T dcc8fde0e9ab4236a33971bbc90716ff P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:23.754297 27599 raft_consensus.cc:348] T 52a4b0ee48204fd18ebb932448bd6503 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:23.754580 27599 raft_consensus.cc:374] T 52a4b0ee48204fd18ebb932448bd6503 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:23.754691 27599 raft_consensus.cc:729] T 52a4b0ee48204fd18ebb932448bd6503 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:23.755012 27599 consensus_queue.cc:260] T 52a4b0ee48204fd18ebb932448bd6503 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:23.756134 27599 ts_tablet_manager.cc:1398] T 52a4b0ee48204fd18ebb932448bd6503 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:23.756513 27599 tablet_bootstrap.cc:492] T 18c4a357e8814da8aac5b526bc920dab P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:23.758678 27599 tablet_bootstrap.cc:654] T 18c4a357e8814da8aac5b526bc920dab P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:23.758915 27597 tablet_bootstrap.cc:492] T 18c4a357e8814da8aac5b526bc920dab P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:23.759091 27597 ts_tablet_manager.cc:1367] T 18c4a357e8814da8aac5b526bc920dab P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240502 14:14:23.760038 27597 raft_consensus.cc:348] T 18c4a357e8814da8aac5b526bc920dab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:23.760327 27597 raft_consensus.cc:374] T 18c4a357e8814da8aac5b526bc920dab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:23.760447 27597 raft_consensus.cc:729] T 18c4a357e8814da8aac5b526bc920dab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:23.760748 27597 consensus_queue.cc:260] T 18c4a357e8814da8aac5b526bc920dab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:23.761616 27597 ts_tablet_manager.cc:1398] T 18c4a357e8814da8aac5b526bc920dab P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240502 14:14:23.762264 27575 raft_consensus.cc:482] T 52a4b0ee48204fd18ebb932448bd6503 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:23.762454 27575 raft_consensus.cc:504] T 52a4b0ee48204fd18ebb932448bd6503 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:23.763442 27575 leader_election.cc:288] T 52a4b0ee48204fd18ebb932448bd6503 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:14:23.764071 27445 raft_consensus.cc:482] T dcc8fde0e9ab4236a33971bbc90716ff P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:23.764273 27445 raft_consensus.cc:504] T dcc8fde0e9ab4236a33971bbc90716ff P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:23.764542 26557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52a4b0ee48204fd18ebb932448bd6503" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:23.764760 26557 raft_consensus.cc:2454] T 52a4b0ee48204fd18ebb932448bd6503 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:14:23.765106 27445 leader_election.cc:288] T dcc8fde0e9ab4236a33971bbc90716ff P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:23.765498 26362 leader_election.cc:302] T 52a4b0ee48204fd18ebb932448bd6503 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:23.766572 26819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc8fde0e9ab4236a33971bbc90716ff" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:14:23.767024 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52a4b0ee48204fd18ebb932448bd6503" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:14:23.767343 26818 raft_consensus.cc:2454] T 52a4b0ee48204fd18ebb932448bd6503 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
W20240502 14:14:23.767459 26494 leader_election.cc:341] T dcc8fde0e9ab4236a33971bbc90716ff P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:23.767763 27575 raft_consensus.cc:2781] T 52a4b0ee48204fd18ebb932448bd6503 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:23.767886 27575 raft_consensus.cc:482] T 52a4b0ee48204fd18ebb932448bd6503 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:23.767975 27575 raft_consensus.cc:3037] T 52a4b0ee48204fd18ebb932448bd6503 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:23.769934 27598 tablet_bootstrap.cc:492] T dcc8fde0e9ab4236a33971bbc90716ff P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:23.770093 27598 ts_tablet_manager.cc:1367] T dcc8fde0e9ab4236a33971bbc90716ff P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.007s
I20240502 14:14:23.770742 27598 raft_consensus.cc:348] T dcc8fde0e9ab4236a33971bbc90716ff P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:23.770912 27598 raft_consensus.cc:374] T dcc8fde0e9ab4236a33971bbc90716ff P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:23.771047 27598 raft_consensus.cc:729] T dcc8fde0e9ab4236a33971bbc90716ff P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:23.771274 27598 consensus_queue.cc:260] T dcc8fde0e9ab4236a33971bbc90716ff P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:23.771835 27575 raft_consensus.cc:504] T 52a4b0ee48204fd18ebb932448bd6503 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:23.772694 27598 ts_tablet_manager.cc:1398] T dcc8fde0e9ab4236a33971bbc90716ff P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:14:23.773252 27575 leader_election.cc:288] T 52a4b0ee48204fd18ebb932448bd6503 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Requested vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:14:23.773252 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52a4b0ee48204fd18ebb932448bd6503" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:14:23.773371 26557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52a4b0ee48204fd18ebb932448bd6503" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:14:23.773439 26818 raft_consensus.cc:3037] T 52a4b0ee48204fd18ebb932448bd6503 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:23.773625 26557 raft_consensus.cc:3037] T 52a4b0ee48204fd18ebb932448bd6503 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:23.776626 26557 raft_consensus.cc:2454] T 52a4b0ee48204fd18ebb932448bd6503 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:14:23.776877 26818 raft_consensus.cc:2454] T 52a4b0ee48204fd18ebb932448bd6503 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:14:23.777074 26362 leader_election.cc:302] T 52a4b0ee48204fd18ebb932448bd6503 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:23.777418 27575 raft_consensus.cc:2781] T 52a4b0ee48204fd18ebb932448bd6503 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:23.777536 27575 raft_consensus.cc:686] T 52a4b0ee48204fd18ebb932448bd6503 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Becoming Leader. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Running, Role: LEADER
I20240502 14:14:23.777837 27575 consensus_queue.cc:237] T 52a4b0ee48204fd18ebb932448bd6503 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:23.780916 26281 catalog_manager.cc:5484] T 52a4b0ee48204fd18ebb932448bd6503 P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: term changed from 0 to 1, leader changed from <none> to 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193). New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:14:23.782269 27599 tablet_bootstrap.cc:492] T 18c4a357e8814da8aac5b526bc920dab P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:23.782433 27599 ts_tablet_manager.cc:1367] T 18c4a357e8814da8aac5b526bc920dab P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.026s user 0.010s sys 0.000s
I20240502 14:14:23.783241 27599 raft_consensus.cc:348] T 18c4a357e8814da8aac5b526bc920dab P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:23.783476 27599 raft_consensus.cc:374] T 18c4a357e8814da8aac5b526bc920dab P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:23.783573 27599 raft_consensus.cc:729] T 18c4a357e8814da8aac5b526bc920dab P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:23.783846 27599 consensus_queue.cc:260] T 18c4a357e8814da8aac5b526bc920dab P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:23.784502 27599 ts_tablet_manager.cc:1398] T 18c4a357e8814da8aac5b526bc920dab P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:23.785480 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc8fde0e9ab4236a33971bbc90716ff" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:14:23.785681 26426 raft_consensus.cc:2454] T dcc8fde0e9ab4236a33971bbc90716ff P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 0.
I20240502 14:14:23.786132 26493 leader_election.cc:302] T dcc8fde0e9ab4236a33971bbc90716ff P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, ba187e38a2754d60a12de9ac4856430a; no voters: e4cfb8fc5a474177af6848a6efa13a27
I20240502 14:14:23.786499 27445 raft_consensus.cc:2781] T dcc8fde0e9ab4236a33971bbc90716ff P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:23.786607 27445 raft_consensus.cc:482] T dcc8fde0e9ab4236a33971bbc90716ff P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:23.786700 27445 raft_consensus.cc:3037] T dcc8fde0e9ab4236a33971bbc90716ff P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:23.788801 27445 raft_consensus.cc:504] T dcc8fde0e9ab4236a33971bbc90716ff P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:23.789386 27445 leader_election.cc:288] T dcc8fde0e9ab4236a33971bbc90716ff P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Requested vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:23.789855 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc8fde0e9ab4236a33971bbc90716ff" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:14:23.790011 26426 raft_consensus.cc:3037] T dcc8fde0e9ab4236a33971bbc90716ff P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:23.789938 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc8fde0e9ab4236a33971bbc90716ff" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:14:23.790171 26818 raft_consensus.cc:3037] T dcc8fde0e9ab4236a33971bbc90716ff P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:23.791932 26426 raft_consensus.cc:2454] T dcc8fde0e9ab4236a33971bbc90716ff P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:14:23.792021 26818 raft_consensus.cc:2454] T dcc8fde0e9ab4236a33971bbc90716ff P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:14:23.792311 26493 leader_election.cc:302] T dcc8fde0e9ab4236a33971bbc90716ff P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:23.792829 27445 raft_consensus.cc:2781] T dcc8fde0e9ab4236a33971bbc90716ff P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:23.792970 27445 raft_consensus.cc:686] T dcc8fde0e9ab4236a33971bbc90716ff P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Becoming Leader. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Running, Role: LEADER
I20240502 14:14:23.793254 27445 consensus_queue.cc:237] T dcc8fde0e9ab4236a33971bbc90716ff P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:23.795715 26281 catalog_manager.cc:5484] T dcc8fde0e9ab4236a33971bbc90716ff P ba187e38a2754d60a12de9ac4856430a reported cstate change: term changed from 0 to 1, leader changed from <none> to ba187e38a2754d60a12de9ac4856430a (127.19.94.194). New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:14:24.020432 27422 raft_consensus.cc:482] T 18c4a357e8814da8aac5b526bc920dab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:24.020678 27422 raft_consensus.cc:504] T 18c4a357e8814da8aac5b526bc920dab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:24.021622 27422 leader_election.cc:288] T 18c4a357e8814da8aac5b526bc920dab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:14:24.022473 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c4a357e8814da8aac5b526bc920dab" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:14:24.022775 26426 raft_consensus.cc:2454] T 18c4a357e8814da8aac5b526bc920dab P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:14:24.023715 26624 leader_election.cc:302] T 18c4a357e8814da8aac5b526bc920dab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:14:24.024080 27422 raft_consensus.cc:2781] T 18c4a357e8814da8aac5b526bc920dab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:24.024199 27422 raft_consensus.cc:482] T 18c4a357e8814da8aac5b526bc920dab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:24.024292 27422 raft_consensus.cc:3037] T 18c4a357e8814da8aac5b526bc920dab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:24.027215 27422 raft_consensus.cc:504] T 18c4a357e8814da8aac5b526bc920dab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:24.028095 27422 leader_election.cc:288] T 18c4a357e8814da8aac5b526bc920dab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Requested vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:14:24.028687 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c4a357e8814da8aac5b526bc920dab" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:14:24.029111 26426 raft_consensus.cc:3037] T 18c4a357e8814da8aac5b526bc920dab P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:24.032002 26426 raft_consensus.cc:2454] T 18c4a357e8814da8aac5b526bc920dab P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:14:24.032151 26819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c4a357e8814da8aac5b526bc920dab" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:14:24.032423 26819 raft_consensus.cc:3037] T 18c4a357e8814da8aac5b526bc920dab P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:24.032557 26624 leader_election.cc:302] T 18c4a357e8814da8aac5b526bc920dab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:14:24.032766 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c4a357e8814da8aac5b526bc920dab" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:14:24.032914 27422 raft_consensus.cc:2781] T 18c4a357e8814da8aac5b526bc920dab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:24.034029 27422 raft_consensus.cc:686] T 18c4a357e8814da8aac5b526bc920dab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Becoming Leader. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Running, Role: LEADER
I20240502 14:14:24.034387 27422 consensus_queue.cc:237] T 18c4a357e8814da8aac5b526bc920dab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:24.035387 26819 raft_consensus.cc:2454] T 18c4a357e8814da8aac5b526bc920dab P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:14:24.038589 26281 catalog_manager.cc:5484] T 18c4a357e8814da8aac5b526bc920dab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195). New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:14:24.184652 27445 consensus_queue.cc:1035] T dcc8fde0e9ab4236a33971bbc90716ff P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:24.195345 27445 consensus_queue.cc:1035] T dcc8fde0e9ab4236a33971bbc90716ff P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:24.356539 27575 consensus_queue.cc:1035] T 52a4b0ee48204fd18ebb932448bd6503 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:24.362553 27624 consensus_queue.cc:1035] T 52a4b0ee48204fd18ebb932448bd6503 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:24.452862 27617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:24.453222 27617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:24.460981 27617 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:14:24.473587 27422 consensus_queue.cc:1035] T 18c4a357e8814da8aac5b526bc920dab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:24.481664 27615 consensus_queue.cc:1035] T 18c4a357e8814da8aac5b526bc920dab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 319.229 ms
time per row: 0.159614 ms
I20240502 14:14:24.939486 26277 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:24.949275 26669 tablet_service.cc:1467] Processing CreateTablet for tablet 66554eb8b44145d4b474707b061c9659 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e814fcdd8ac94d6eb54d1c57f004c870]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:24.949795 26669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66554eb8b44145d4b474707b061c9659. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:24.949704 26407 tablet_service.cc:1467] Processing CreateTablet for tablet 66554eb8b44145d4b474707b061c9659 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e814fcdd8ac94d6eb54d1c57f004c870]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:24.950150 26407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66554eb8b44145d4b474707b061c9659. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:24.950050 26797 tablet_service.cc:1467] Processing CreateTablet for tablet 39a61744642f40c8ae92da708dc91ca3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e814fcdd8ac94d6eb54d1c57f004c870]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:24.950459 26797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39a61744642f40c8ae92da708dc91ca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:24.951484 26799 tablet_service.cc:1467] Processing CreateTablet for tablet c586354e10cb4bedbbb20bd103ad477e (DEFAULT_TABLE table=rebalance_test_table_3 [id=e814fcdd8ac94d6eb54d1c57f004c870]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:24.951910 26799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c586354e10cb4bedbbb20bd103ad477e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:24.957545 26668 tablet_service.cc:1467] Processing CreateTablet for tablet 39a61744642f40c8ae92da708dc91ca3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e814fcdd8ac94d6eb54d1c57f004c870]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:24.958019 26668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39a61744642f40c8ae92da708dc91ca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:24.958271 26536 tablet_service.cc:1467] Processing CreateTablet for tablet 66554eb8b44145d4b474707b061c9659 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e814fcdd8ac94d6eb54d1c57f004c870]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:24.958890 26536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66554eb8b44145d4b474707b061c9659. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:24.960716 27687 tablet_bootstrap.cc:492] T 66554eb8b44145d4b474707b061c9659 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:24.961061 26667 tablet_service.cc:1467] Processing CreateTablet for tablet c586354e10cb4bedbbb20bd103ad477e (DEFAULT_TABLE table=rebalance_test_table_3 [id=e814fcdd8ac94d6eb54d1c57f004c870]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:24.961239 26537 tablet_service.cc:1467] Processing CreateTablet for tablet c586354e10cb4bedbbb20bd103ad477e (DEFAULT_TABLE table=rebalance_test_table_3 [id=e814fcdd8ac94d6eb54d1c57f004c870]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:24.961544 26667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c586354e10cb4bedbbb20bd103ad477e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:24.961758 26537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c586354e10cb4bedbbb20bd103ad477e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:24.962823 27687 tablet_bootstrap.cc:654] T 66554eb8b44145d4b474707b061c9659 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:24.965449 26405 tablet_service.cc:1467] Processing CreateTablet for tablet 39a61744642f40c8ae92da708dc91ca3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e814fcdd8ac94d6eb54d1c57f004c870]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:24.966094 26405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39a61744642f40c8ae92da708dc91ca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:24.971485 27687 tablet_bootstrap.cc:492] T 66554eb8b44145d4b474707b061c9659 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:24.971776 27687 ts_tablet_manager.cc:1367] T 66554eb8b44145d4b474707b061c9659 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20240502 14:14:24.973477 27687 raft_consensus.cc:348] T 66554eb8b44145d4b474707b061c9659 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:24.973821 27687 raft_consensus.cc:374] T 66554eb8b44145d4b474707b061c9659 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:24.973976 27687 raft_consensus.cc:729] T 66554eb8b44145d4b474707b061c9659 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:24.974310 27687 consensus_queue.cc:260] T 66554eb8b44145d4b474707b061c9659 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:24.975062 27687 ts_tablet_manager.cc:1398] T 66554eb8b44145d4b474707b061c9659 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:24.976801 27687 tablet_bootstrap.cc:492] T c586354e10cb4bedbbb20bd103ad477e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:24.982061 27687 tablet_bootstrap.cc:654] T c586354e10cb4bedbbb20bd103ad477e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:24.989320 27692 tablet_bootstrap.cc:492] T 39a61744642f40c8ae92da708dc91ca3 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:24.992213 27692 tablet_bootstrap.cc:654] T 39a61744642f40c8ae92da708dc91ca3 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:24.995090 27690 tablet_bootstrap.cc:492] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:24.998642 27690 tablet_bootstrap.cc:654] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:25.000741 27688 tablet_bootstrap.cc:492] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:25.002579 27692 tablet_bootstrap.cc:492] T 39a61744642f40c8ae92da708dc91ca3 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:25.002588 27687 tablet_bootstrap.cc:492] T c586354e10cb4bedbbb20bd103ad477e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:25.002782 27687 ts_tablet_manager.cc:1367] T c586354e10cb4bedbbb20bd103ad477e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.026s user 0.013s sys 0.008s
I20240502 14:14:25.002842 27692 ts_tablet_manager.cc:1367] T 39a61744642f40c8ae92da708dc91ca3 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.005s
I20240502 14:14:25.003401 27688 tablet_bootstrap.cc:654] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:25.003600 27692 raft_consensus.cc:348] T 39a61744642f40c8ae92da708dc91ca3 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:25.003798 27692 raft_consensus.cc:374] T 39a61744642f40c8ae92da708dc91ca3 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:25.003706 27687 raft_consensus.cc:348] T c586354e10cb4bedbbb20bd103ad477e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:25.003928 27692 raft_consensus.cc:729] T 39a61744642f40c8ae92da708dc91ca3 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:25.004005 27687 raft_consensus.cc:374] T c586354e10cb4bedbbb20bd103ad477e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:25.004168 27687 raft_consensus.cc:729] T c586354e10cb4bedbbb20bd103ad477e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:25.004211 27692 consensus_queue.cc:260] T 39a61744642f40c8ae92da708dc91ca3 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:25.005324 27692 ts_tablet_manager.cc:1398] T 39a61744642f40c8ae92da708dc91ca3 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:14:25.005642 27690 tablet_bootstrap.cc:492] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:25.005692 27692 tablet_bootstrap.cc:492] T 66554eb8b44145d4b474707b061c9659 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:25.004640 27687 consensus_queue.cc:260] T c586354e10cb4bedbbb20bd103ad477e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:25.005872 27690 ts_tablet_manager.cc:1367] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240502 14:14:25.006071 27422 raft_consensus.cc:482] T c586354e10cb4bedbbb20bd103ad477e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:25.006271 27422 raft_consensus.cc:504] T c586354e10cb4bedbbb20bd103ad477e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:25.007750 27422 leader_election.cc:288] T c586354e10cb4bedbbb20bd103ad477e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:25.008494 27690 raft_consensus.cc:348] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:25.008796 27690 raft_consensus.cc:374] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:25.008734 26817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c586354e10cb4bedbbb20bd103ad477e" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:14:25.008793 26557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c586354e10cb4bedbbb20bd103ad477e" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:25.008966 27690 raft_consensus.cc:729] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
W20240502 14:14:25.009721 26625 leader_election.cc:341] T c586354e10cb4bedbbb20bd103ad477e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:25.009902 27692 tablet_bootstrap.cc:654] T 66554eb8b44145d4b474707b061c9659 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
W20240502 14:14:25.010448 26624 leader_election.cc:341] T c586354e10cb4bedbbb20bd103ad477e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:25.010596 26624 leader_election.cc:302] T c586354e10cb4bedbbb20bd103ad477e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters: ba187e38a2754d60a12de9ac4856430a, e4cfb8fc5a474177af6848a6efa13a27
I20240502 14:14:25.011339 27422 raft_consensus.cc:2726] T c586354e10cb4bedbbb20bd103ad477e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:14:25.012522 27687 ts_tablet_manager.cc:1398] T c586354e10cb4bedbbb20bd103ad477e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20240502 14:14:25.013036 27687 tablet_bootstrap.cc:492] T 39a61744642f40c8ae92da708dc91ca3 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:25.009234 27690 consensus_queue.cc:260] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:25.017572 27687 tablet_bootstrap.cc:654] T 39a61744642f40c8ae92da708dc91ca3 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:25.020954 27687 tablet_bootstrap.cc:492] T 39a61744642f40c8ae92da708dc91ca3 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:25.020954 27688 tablet_bootstrap.cc:492] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:25.021216 27687 ts_tablet_manager.cc:1367] T 39a61744642f40c8ae92da708dc91ca3 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.001s
I20240502 14:14:25.021282 27688 ts_tablet_manager.cc:1367] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.008s
I20240502 14:14:25.021965 27692 tablet_bootstrap.cc:492] T 66554eb8b44145d4b474707b061c9659 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:25.022264 27692 ts_tablet_manager.cc:1367] T 66554eb8b44145d4b474707b061c9659 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.007s
I20240502 14:14:25.022204 27688 raft_consensus.cc:348] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:25.022174 27687 raft_consensus.cc:348] T 39a61744642f40c8ae92da708dc91ca3 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:25.022410 27687 raft_consensus.cc:374] T 39a61744642f40c8ae92da708dc91ca3 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:25.022524 27688 raft_consensus.cc:374] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:25.022521 27687 raft_consensus.cc:729] T 39a61744642f40c8ae92da708dc91ca3 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:25.022732 27688 raft_consensus.cc:729] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:25.022806 27687 consensus_queue.cc:260] T 39a61744642f40c8ae92da708dc91ca3 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:25.023238 27692 raft_consensus.cc:348] T 66554eb8b44145d4b474707b061c9659 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:25.023434 27687 ts_tablet_manager.cc:1398] T 39a61744642f40c8ae92da708dc91ca3 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:14:25.023545 27692 raft_consensus.cc:374] T 66554eb8b44145d4b474707b061c9659 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:25.023711 27692 raft_consensus.cc:729] T 66554eb8b44145d4b474707b061c9659 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:25.023747 27688 consensus_queue.cc:260] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:25.024240 27692 consensus_queue.cc:260] T 66554eb8b44145d4b474707b061c9659 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:25.024462 27688 ts_tablet_manager.cc:1398] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:14:25.024817 27688 tablet_bootstrap.cc:492] T c586354e10cb4bedbbb20bd103ad477e P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:25.025686 27692 ts_tablet_manager.cc:1398] T 66554eb8b44145d4b474707b061c9659 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:25.026517 27688 tablet_bootstrap.cc:654] T c586354e10cb4bedbbb20bd103ad477e P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:25.031330 27688 tablet_bootstrap.cc:492] T c586354e10cb4bedbbb20bd103ad477e P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:25.031526 27688 ts_tablet_manager.cc:1367] T c586354e10cb4bedbbb20bd103ad477e P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.002s
I20240502 14:14:25.032275 27688 raft_consensus.cc:348] T c586354e10cb4bedbbb20bd103ad477e P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:25.032541 27688 raft_consensus.cc:374] T c586354e10cb4bedbbb20bd103ad477e P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:25.032625 27688 raft_consensus.cc:729] T c586354e10cb4bedbbb20bd103ad477e P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:25.032855 27690 ts_tablet_manager.cc:1398] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20240502 14:14:25.032886 27688 consensus_queue.cc:260] T c586354e10cb4bedbbb20bd103ad477e P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:25.033241 27690 tablet_bootstrap.cc:492] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:25.033558 27688 ts_tablet_manager.cc:1398] T c586354e10cb4bedbbb20bd103ad477e P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:25.036540 27690 tablet_bootstrap.cc:654] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:25.038987 27690 tablet_bootstrap.cc:492] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:25.039189 27690 ts_tablet_manager.cc:1367] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20240502 14:14:25.039988 27690 raft_consensus.cc:348] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:25.040237 27690 raft_consensus.cc:374] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:25.040310 27690 raft_consensus.cc:729] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:25.040611 27690 consensus_queue.cc:260] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:25.041234 27690 ts_tablet_manager.cc:1398] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:14:25.107874 27442 raft_consensus.cc:482] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:25.108099 27442 raft_consensus.cc:504] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:25.109043 27442 leader_election.cc:288] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:14:25.109750 26427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39a61744642f40c8ae92da708dc91ca3" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:14:25.110090 26427 raft_consensus.cc:2454] T 39a61744642f40c8ae92da708dc91ca3 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 0.
I20240502 14:14:25.110633 26754 leader_election.cc:302] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:25.110962 27442 raft_consensus.cc:2781] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:25.111126 27442 raft_consensus.cc:482] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:25.111277 27442 raft_consensus.cc:3037] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:25.111356 27580 raft_consensus.cc:482] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:25.111515 27580 raft_consensus.cc:504] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:25.112293 27580 leader_election.cc:288] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:14:25.113044 26427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66554eb8b44145d4b474707b061c9659" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:14:25.113296 26427 raft_consensus.cc:2454] T 66554eb8b44145d4b474707b061c9659 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 0.
I20240502 14:14:25.113253 26689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66554eb8b44145d4b474707b061c9659" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:14:25.113459 26689 raft_consensus.cc:2454] T 66554eb8b44145d4b474707b061c9659 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 0.
I20240502 14:14:25.113787 26493 leader_election.cc:302] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:25.114060 27580 raft_consensus.cc:2781] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:25.114185 27580 raft_consensus.cc:482] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:25.114281 27580 raft_consensus.cc:3037] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:25.114392 27442 raft_consensus.cc:504] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:25.115309 27442 leader_election.cc:288] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Requested vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:14:25.115641 26427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39a61744642f40c8ae92da708dc91ca3" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:14:25.115846 26427 raft_consensus.cc:3037] T 39a61744642f40c8ae92da708dc91ca3 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:25.117646 26689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39a61744642f40c8ae92da708dc91ca3" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:14:25.117110 27580 raft_consensus.cc:504] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:25.117918 26689 raft_consensus.cc:2454] T 39a61744642f40c8ae92da708dc91ca3 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 0.
I20240502 14:14:25.117729 26687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39a61744642f40c8ae92da708dc91ca3" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:14:25.118247 26687 raft_consensus.cc:3037] T 39a61744642f40c8ae92da708dc91ca3 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:25.118849 26427 raft_consensus.cc:2454] T 39a61744642f40c8ae92da708dc91ca3 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
I20240502 14:14:25.119096 27580 leader_election.cc:288] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Requested vote from peers 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:14:25.119455 26754 leader_election.cc:302] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:25.119781 27442 raft_consensus.cc:2781] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:25.119742 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66554eb8b44145d4b474707b061c9659" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:14:25.119937 26426 raft_consensus.cc:3037] T 66554eb8b44145d4b474707b061c9659 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:25.120035 26688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66554eb8b44145d4b474707b061c9659" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:14:25.120267 26688 raft_consensus.cc:3037] T 66554eb8b44145d4b474707b061c9659 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:25.121387 26687 raft_consensus.cc:2454] T 39a61744642f40c8ae92da708dc91ca3 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
I20240502 14:14:25.122560 27442 raft_consensus.cc:686] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Becoming Leader. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Running, Role: LEADER
I20240502 14:14:25.122915 27442 consensus_queue.cc:237] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:25.123102 26688 raft_consensus.cc:2454] T 66554eb8b44145d4b474707b061c9659 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:14:25.123251 26426 raft_consensus.cc:2454] T 66554eb8b44145d4b474707b061c9659 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:14:25.123589 26494 leader_election.cc:302] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:25.123950 27580 raft_consensus.cc:2781] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:25.124112 27580 raft_consensus.cc:686] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Becoming Leader. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Running, Role: LEADER
I20240502 14:14:25.124518 27580 consensus_queue.cc:237] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:25.126745 26277 catalog_manager.cc:5484] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196). New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:14:25.130723 26280 catalog_manager.cc:5484] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a reported cstate change: term changed from 0 to 1, leader changed from <none> to ba187e38a2754d60a12de9ac4856430a (127.19.94.194). New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:14:25.214705 27580 raft_consensus.cc:482] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:25.214926 27580 raft_consensus.cc:504] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:25.215715 27580 leader_election.cc:288] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:25.216313 26688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c586354e10cb4bedbbb20bd103ad477e" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:14:25.216423 26817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c586354e10cb4bedbbb20bd103ad477e" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:14:25.216614 26688 raft_consensus.cc:2454] T c586354e10cb4bedbbb20bd103ad477e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 0.
I20240502 14:14:25.216668 26817 raft_consensus.cc:2454] T c586354e10cb4bedbbb20bd103ad477e P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 0.
I20240502 14:14:25.217104 26494 leader_election.cc:302] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:25.217736 27580 raft_consensus.cc:2781] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:25.217868 27580 raft_consensus.cc:482] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:25.217993 27580 raft_consensus.cc:3037] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:25.221248 27580 raft_consensus.cc:504] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:25.222116 27580 leader_election.cc:288] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Requested vote from peers 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:25.222725 26817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c586354e10cb4bedbbb20bd103ad477e" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:14:25.222779 26688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c586354e10cb4bedbbb20bd103ad477e" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:14:25.222972 26817 raft_consensus.cc:3037] T c586354e10cb4bedbbb20bd103ad477e P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:25.223014 26688 raft_consensus.cc:3037] T c586354e10cb4bedbbb20bd103ad477e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:25.226096 26817 raft_consensus.cc:2454] T c586354e10cb4bedbbb20bd103ad477e P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:14:25.226384 26688 raft_consensus.cc:2454] T c586354e10cb4bedbbb20bd103ad477e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:14:25.226976 26494 leader_election.cc:302] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:25.227316 27580 raft_consensus.cc:2781] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:25.227540 27580 raft_consensus.cc:686] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Becoming Leader. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Running, Role: LEADER
I20240502 14:14:25.227897 27580 consensus_queue.cc:237] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:25.231957 26280 catalog_manager.cc:5484] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a reported cstate change: term changed from 0 to 1, leader changed from <none> to ba187e38a2754d60a12de9ac4856430a (127.19.94.194). New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:14:25.523139 27580 consensus_queue.cc:1035] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:25.529230 27445 consensus_queue.cc:1035] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:25.594532 27442 consensus_queue.cc:1035] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:25.600564 27701 consensus_queue.cc:1035] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:25.640676 27705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:25.641055 27705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:25.649628 27705 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240502 14:14:25.809101 26688 raft_consensus.cc:1261] T c586354e10cb4bedbbb20bd103ad477e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:25.809127 26817 raft_consensus.cc:1261] T c586354e10cb4bedbbb20bd103ad477e P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:25.817808 27580 consensus_queue.cc:1035] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:25.818490 27580 consensus_queue.cc:1035] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:25.982981 27710 mvcc.cc:205] Tried to move back new op lower bound from 7023244352768094208 to 7023244350376321024. Current Snapshot: MvccSnapshot[applied={T|T < 7023244352739934208}]
INSERT report
rows total: 2000
time total: 272.586 ms
time per row: 0.136293 ms
I20240502 14:14:26.081765 26280 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:26.091076 26536 tablet_service.cc:1467] Processing CreateTablet for tablet a7aae43aeafc491b84d879fecfbd983f (DEFAULT_TABLE table=rebalance_test_table_4 [id=10659483777045cb826c0f72b9086a71]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:26.091323 26405 tablet_service.cc:1467] Processing CreateTablet for tablet a7aae43aeafc491b84d879fecfbd983f (DEFAULT_TABLE table=rebalance_test_table_4 [id=10659483777045cb826c0f72b9086a71]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:26.091626 26536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7aae43aeafc491b84d879fecfbd983f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:26.091652 26668 tablet_service.cc:1467] Processing CreateTablet for tablet a7aae43aeafc491b84d879fecfbd983f (DEFAULT_TABLE table=rebalance_test_table_4 [id=10659483777045cb826c0f72b9086a71]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:26.091778 26405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7aae43aeafc491b84d879fecfbd983f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:26.092131 26668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7aae43aeafc491b84d879fecfbd983f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:26.096719 27768 tablet_bootstrap.cc:492] T a7aae43aeafc491b84d879fecfbd983f P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:26.097309 26407 tablet_service.cc:1467] Processing CreateTablet for tablet 8a8bd7225f5c48feb980049b99eb99c1 (DEFAULT_TABLE table=rebalance_test_table_4 [id=10659483777045cb826c0f72b9086a71]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:26.098076 26407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a8bd7225f5c48feb980049b99eb99c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:26.098944 27768 tablet_bootstrap.cc:654] T a7aae43aeafc491b84d879fecfbd983f P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:26.100210 26797 tablet_service.cc:1467] Processing CreateTablet for tablet 8a8bd7225f5c48feb980049b99eb99c1 (DEFAULT_TABLE table=rebalance_test_table_4 [id=10659483777045cb826c0f72b9086a71]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:26.100745 27770 tablet_bootstrap.cc:492] T a7aae43aeafc491b84d879fecfbd983f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:26.100766 26797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a8bd7225f5c48feb980049b99eb99c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:26.103122 27770 tablet_bootstrap.cc:654] T a7aae43aeafc491b84d879fecfbd983f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:26.109125 26537 tablet_service.cc:1467] Processing CreateTablet for tablet 8a8bd7225f5c48feb980049b99eb99c1 (DEFAULT_TABLE table=rebalance_test_table_4 [id=10659483777045cb826c0f72b9086a71]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:26.109830 26537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a8bd7225f5c48feb980049b99eb99c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:26.110713 26667 tablet_service.cc:1467] Processing CreateTablet for tablet b8dd27ac333542a4844172fdc8e9b9bd (DEFAULT_TABLE table=rebalance_test_table_4 [id=10659483777045cb826c0f72b9086a71]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:26.111320 26667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8dd27ac333542a4844172fdc8e9b9bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:26.114471 27770 tablet_bootstrap.cc:492] T a7aae43aeafc491b84d879fecfbd983f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:26.114583 27768 tablet_bootstrap.cc:492] T a7aae43aeafc491b84d879fecfbd983f P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:26.114734 27770 ts_tablet_manager.cc:1367] T a7aae43aeafc491b84d879fecfbd983f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.003s
I20240502 14:14:26.114770 27768 ts_tablet_manager.cc:1367] T a7aae43aeafc491b84d879fecfbd983f P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.004s
I20240502 14:14:26.115667 27768 raft_consensus.cc:348] T a7aae43aeafc491b84d879fecfbd983f P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:26.115674 27770 raft_consensus.cc:348] T a7aae43aeafc491b84d879fecfbd983f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:26.115893 27768 raft_consensus.cc:374] T a7aae43aeafc491b84d879fecfbd983f P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:26.115983 27768 raft_consensus.cc:729] T a7aae43aeafc491b84d879fecfbd983f P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:26.115985 27770 raft_consensus.cc:374] T a7aae43aeafc491b84d879fecfbd983f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:26.116151 27770 raft_consensus.cc:729] T a7aae43aeafc491b84d879fecfbd983f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:26.116254 27768 consensus_queue.cc:260] T a7aae43aeafc491b84d879fecfbd983f P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:26.116484 27770 consensus_queue.cc:260] T a7aae43aeafc491b84d879fecfbd983f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:26.116945 27768 ts_tablet_manager.cc:1398] T a7aae43aeafc491b84d879fecfbd983f P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:26.117146 27770 ts_tablet_manager.cc:1398] T a7aae43aeafc491b84d879fecfbd983f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:26.117323 27768 tablet_bootstrap.cc:492] T 8a8bd7225f5c48feb980049b99eb99c1 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:26.117779 26799 tablet_service.cc:1467] Processing CreateTablet for tablet b8dd27ac333542a4844172fdc8e9b9bd (DEFAULT_TABLE table=rebalance_test_table_4 [id=10659483777045cb826c0f72b9086a71]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:26.118237 26799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8dd27ac333542a4844172fdc8e9b9bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:26.121454 27768 tablet_bootstrap.cc:654] T 8a8bd7225f5c48feb980049b99eb99c1 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:26.125279 27768 tablet_bootstrap.cc:492] T 8a8bd7225f5c48feb980049b99eb99c1 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:26.125509 27768 ts_tablet_manager.cc:1367] T 8a8bd7225f5c48feb980049b99eb99c1 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240502 14:14:26.126477 27768 raft_consensus.cc:348] T 8a8bd7225f5c48feb980049b99eb99c1 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:26.126742 27768 raft_consensus.cc:374] T 8a8bd7225f5c48feb980049b99eb99c1 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:26.126822 27768 raft_consensus.cc:729] T 8a8bd7225f5c48feb980049b99eb99c1 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:26.127107 27768 consensus_queue.cc:260] T 8a8bd7225f5c48feb980049b99eb99c1 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:26.127903 27768 ts_tablet_manager.cc:1398] T 8a8bd7225f5c48feb980049b99eb99c1 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 14:14:26.127729 26407 tablet_service.cc:1467] Processing CreateTablet for tablet b8dd27ac333542a4844172fdc8e9b9bd (DEFAULT_TABLE table=rebalance_test_table_4 [id=10659483777045cb826c0f72b9086a71]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:26.128545 26407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8dd27ac333542a4844172fdc8e9b9bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:26.129877 27770 tablet_bootstrap.cc:492] T b8dd27ac333542a4844172fdc8e9b9bd P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:26.131855 27770 tablet_bootstrap.cc:654] T b8dd27ac333542a4844172fdc8e9b9bd P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:26.138947 27770 tablet_bootstrap.cc:492] T b8dd27ac333542a4844172fdc8e9b9bd P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:26.139334 27770 ts_tablet_manager.cc:1367] T b8dd27ac333542a4844172fdc8e9b9bd P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20240502 14:14:26.139524 27774 tablet_bootstrap.cc:492] T 8a8bd7225f5c48feb980049b99eb99c1 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:26.140137 27770 raft_consensus.cc:348] T b8dd27ac333542a4844172fdc8e9b9bd P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:26.140340 27770 raft_consensus.cc:374] T b8dd27ac333542a4844172fdc8e9b9bd P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:26.140439 27770 raft_consensus.cc:729] T b8dd27ac333542a4844172fdc8e9b9bd P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:26.140723 27770 consensus_queue.cc:260] T b8dd27ac333542a4844172fdc8e9b9bd P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:26.141250 27770 ts_tablet_manager.cc:1398] T b8dd27ac333542a4844172fdc8e9b9bd P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:14:26.141486 27774 tablet_bootstrap.cc:654] T 8a8bd7225f5c48feb980049b99eb99c1 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:26.141713 27769 tablet_bootstrap.cc:492] T a7aae43aeafc491b84d879fecfbd983f P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:26.143585 27769 tablet_bootstrap.cc:654] T a7aae43aeafc491b84d879fecfbd983f P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:26.154726 27768 tablet_bootstrap.cc:492] T b8dd27ac333542a4844172fdc8e9b9bd P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:26.156638 27774 tablet_bootstrap.cc:492] T 8a8bd7225f5c48feb980049b99eb99c1 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:26.156685 27769 tablet_bootstrap.cc:492] T a7aae43aeafc491b84d879fecfbd983f P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:26.156828 27768 tablet_bootstrap.cc:654] T b8dd27ac333542a4844172fdc8e9b9bd P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:26.156883 27769 ts_tablet_manager.cc:1367] T a7aae43aeafc491b84d879fecfbd983f P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.005s
I20240502 14:14:26.157017 27774 ts_tablet_manager.cc:1367] T 8a8bd7225f5c48feb980049b99eb99c1 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.017s
I20240502 14:14:26.158046 27774 raft_consensus.cc:348] T 8a8bd7225f5c48feb980049b99eb99c1 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:26.157967 27769 raft_consensus.cc:348] T a7aae43aeafc491b84d879fecfbd983f P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:26.158236 27774 raft_consensus.cc:374] T 8a8bd7225f5c48feb980049b99eb99c1 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:26.158243 27769 raft_consensus.cc:374] T a7aae43aeafc491b84d879fecfbd983f P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:26.158344 27769 raft_consensus.cc:729] T a7aae43aeafc491b84d879fecfbd983f P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:26.158327 27774 raft_consensus.cc:729] T 8a8bd7225f5c48feb980049b99eb99c1 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:26.158609 27769 consensus_queue.cc:260] T a7aae43aeafc491b84d879fecfbd983f P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:26.158670 27774 consensus_queue.cc:260] T 8a8bd7225f5c48feb980049b99eb99c1 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:26.159212 27768 tablet_bootstrap.cc:492] T b8dd27ac333542a4844172fdc8e9b9bd P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:26.159282 27769 ts_tablet_manager.cc:1398] T a7aae43aeafc491b84d879fecfbd983f P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:26.159372 27768 ts_tablet_manager.cc:1367] T b8dd27ac333542a4844172fdc8e9b9bd P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 14:14:26.159570 27769 tablet_bootstrap.cc:492] T 8a8bd7225f5c48feb980049b99eb99c1 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:26.160202 27774 ts_tablet_manager.cc:1398] T 8a8bd7225f5c48feb980049b99eb99c1 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:14:26.160523 27774 tablet_bootstrap.cc:492] T b8dd27ac333542a4844172fdc8e9b9bd P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:26.161090 27768 raft_consensus.cc:348] T b8dd27ac333542a4844172fdc8e9b9bd P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:26.161326 27768 raft_consensus.cc:374] T b8dd27ac333542a4844172fdc8e9b9bd P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:26.161332 27769 tablet_bootstrap.cc:654] T 8a8bd7225f5c48feb980049b99eb99c1 P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:26.161409 27768 raft_consensus.cc:729] T b8dd27ac333542a4844172fdc8e9b9bd P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:26.161674 27768 consensus_queue.cc:260] T b8dd27ac333542a4844172fdc8e9b9bd P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:26.162256 27768 ts_tablet_manager.cc:1398] T b8dd27ac333542a4844172fdc8e9b9bd P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:14:26.162916 27774 tablet_bootstrap.cc:654] T b8dd27ac333542a4844172fdc8e9b9bd P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:26.170905 27769 tablet_bootstrap.cc:492] T 8a8bd7225f5c48feb980049b99eb99c1 P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:26.171020 27774 tablet_bootstrap.cc:492] T b8dd27ac333542a4844172fdc8e9b9bd P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:26.171105 27769 ts_tablet_manager.cc:1367] T 8a8bd7225f5c48feb980049b99eb99c1 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20240502 14:14:26.171205 27774 ts_tablet_manager.cc:1367] T b8dd27ac333542a4844172fdc8e9b9bd P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240502 14:14:26.171794 27769 raft_consensus.cc:348] T 8a8bd7225f5c48feb980049b99eb99c1 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:26.171830 27774 raft_consensus.cc:348] T b8dd27ac333542a4844172fdc8e9b9bd P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:26.171994 27774 raft_consensus.cc:374] T b8dd27ac333542a4844172fdc8e9b9bd P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:26.171991 27769 raft_consensus.cc:374] T 8a8bd7225f5c48feb980049b99eb99c1 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:26.172076 27769 raft_consensus.cc:729] T 8a8bd7225f5c48feb980049b99eb99c1 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:26.172072 27774 raft_consensus.cc:729] T b8dd27ac333542a4844172fdc8e9b9bd P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:26.172317 27774 consensus_queue.cc:260] T b8dd27ac333542a4844172fdc8e9b9bd P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:26.172333 27769 consensus_queue.cc:260] T 8a8bd7225f5c48feb980049b99eb99c1 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:26.173055 27769 ts_tablet_manager.cc:1398] T 8a8bd7225f5c48feb980049b99eb99c1 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:26.173422 27774 ts_tablet_manager.cc:1398] T b8dd27ac333542a4844172fdc8e9b9bd P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:26.196125 27575 raft_consensus.cc:482] T b8dd27ac333542a4844172fdc8e9b9bd P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:26.196327 27575 raft_consensus.cc:504] T b8dd27ac333542a4844172fdc8e9b9bd P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:26.197026 27575 leader_election.cc:288] T b8dd27ac333542a4844172fdc8e9b9bd P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:26.197773 26817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8dd27ac333542a4844172fdc8e9b9bd" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:14:26.197825 26689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8dd27ac333542a4844172fdc8e9b9bd" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:14:26.198064 26689 raft_consensus.cc:2454] T b8dd27ac333542a4844172fdc8e9b9bd P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:14:26.198087 26817 raft_consensus.cc:2454] T b8dd27ac333542a4844172fdc8e9b9bd P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:14:26.198599 26363 leader_election.cc:302] T b8dd27ac333542a4844172fdc8e9b9bd P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:14:26.199116 27575 raft_consensus.cc:2781] T b8dd27ac333542a4844172fdc8e9b9bd P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:26.199263 27575 raft_consensus.cc:482] T b8dd27ac333542a4844172fdc8e9b9bd P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:26.199362 27575 raft_consensus.cc:3037] T b8dd27ac333542a4844172fdc8e9b9bd P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:26.201458 27575 raft_consensus.cc:504] T b8dd27ac333542a4844172fdc8e9b9bd P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:26.202117 27575 leader_election.cc:288] T b8dd27ac333542a4844172fdc8e9b9bd P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Requested vote from peers 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:26.202814 26689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8dd27ac333542a4844172fdc8e9b9bd" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:14:26.202849 26817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8dd27ac333542a4844172fdc8e9b9bd" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:14:26.203023 26689 raft_consensus.cc:3037] T b8dd27ac333542a4844172fdc8e9b9bd P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:26.203104 26817 raft_consensus.cc:3037] T b8dd27ac333542a4844172fdc8e9b9bd P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:26.205127 26689 raft_consensus.cc:2454] T b8dd27ac333542a4844172fdc8e9b9bd P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:14:26.205652 26363 leader_election.cc:302] T b8dd27ac333542a4844172fdc8e9b9bd P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:14:26.206013 27575 raft_consensus.cc:2781] T b8dd27ac333542a4844172fdc8e9b9bd P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:26.206071 26817 raft_consensus.cc:2454] T b8dd27ac333542a4844172fdc8e9b9bd P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:14:26.206514 27575 raft_consensus.cc:686] T b8dd27ac333542a4844172fdc8e9b9bd P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Becoming Leader. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Running, Role: LEADER
I20240502 14:14:26.206858 27575 consensus_queue.cc:237] T b8dd27ac333542a4844172fdc8e9b9bd P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:26.210074 26280 catalog_manager.cc:5484] T b8dd27ac333542a4844172fdc8e9b9bd P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: term changed from 0 to 1, leader changed from <none> to 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193). New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:14:26.259573 27422 raft_consensus.cc:482] T a7aae43aeafc491b84d879fecfbd983f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:26.259804 27422 raft_consensus.cc:504] T a7aae43aeafc491b84d879fecfbd983f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:26.260636 27422 leader_election.cc:288] T a7aae43aeafc491b84d879fecfbd983f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:14:26.261085 26557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aae43aeafc491b84d879fecfbd983f" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:26.261148 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aae43aeafc491b84d879fecfbd983f" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:14:26.261338 26557 raft_consensus.cc:2454] T a7aae43aeafc491b84d879fecfbd983f P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:14:26.261416 26426 raft_consensus.cc:2454] T a7aae43aeafc491b84d879fecfbd983f P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:14:26.261766 26624 leader_election.cc:302] T a7aae43aeafc491b84d879fecfbd983f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:26.262004 27422 raft_consensus.cc:2781] T a7aae43aeafc491b84d879fecfbd983f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:26.262130 27422 raft_consensus.cc:482] T a7aae43aeafc491b84d879fecfbd983f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:26.262225 27422 raft_consensus.cc:3037] T a7aae43aeafc491b84d879fecfbd983f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:26.264245 27422 raft_consensus.cc:504] T a7aae43aeafc491b84d879fecfbd983f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:26.264853 27422 leader_election.cc:288] T a7aae43aeafc491b84d879fecfbd983f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Requested vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:14:26.265290 26557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aae43aeafc491b84d879fecfbd983f" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:14:26.265496 26557 raft_consensus.cc:3037] T a7aae43aeafc491b84d879fecfbd983f P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:26.265419 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7aae43aeafc491b84d879fecfbd983f" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:14:26.265695 26426 raft_consensus.cc:3037] T a7aae43aeafc491b84d879fecfbd983f P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:26.267958 26557 raft_consensus.cc:2454] T a7aae43aeafc491b84d879fecfbd983f P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:14:26.268296 26426 raft_consensus.cc:2454] T a7aae43aeafc491b84d879fecfbd983f P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:14:26.268700 27575 raft_consensus.cc:482] T 8a8bd7225f5c48feb980049b99eb99c1 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:26.268883 26624 leader_election.cc:302] T a7aae43aeafc491b84d879fecfbd983f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:26.268898 27575 raft_consensus.cc:504] T 8a8bd7225f5c48feb980049b99eb99c1 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:26.269187 27422 raft_consensus.cc:2781] T a7aae43aeafc491b84d879fecfbd983f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:26.269662 27422 raft_consensus.cc:686] T a7aae43aeafc491b84d879fecfbd983f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Becoming Leader. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Running, Role: LEADER
I20240502 14:14:26.270069 26817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8bd7225f5c48feb980049b99eb99c1" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:14:26.270015 27422 consensus_queue.cc:237] T a7aae43aeafc491b84d879fecfbd983f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:26.270299 26817 raft_consensus.cc:2454] T 8a8bd7225f5c48feb980049b99eb99c1 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:14:26.270759 26363 leader_election.cc:302] T 8a8bd7225f5c48feb980049b99eb99c1 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:26.271037 27781 raft_consensus.cc:2781] T 8a8bd7225f5c48feb980049b99eb99c1 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:26.271144 27781 raft_consensus.cc:482] T 8a8bd7225f5c48feb980049b99eb99c1 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:26.271384 27781 raft_consensus.cc:3037] T 8a8bd7225f5c48feb980049b99eb99c1 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:26.272006 26557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8bd7225f5c48feb980049b99eb99c1" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:26.272228 26557 raft_consensus.cc:2454] T 8a8bd7225f5c48feb980049b99eb99c1 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:14:26.272851 27575 leader_election.cc:288] T 8a8bd7225f5c48feb980049b99eb99c1 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:14:26.274446 27781 raft_consensus.cc:504] T 8a8bd7225f5c48feb980049b99eb99c1 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:26.275095 27781 leader_election.cc:288] T 8a8bd7225f5c48feb980049b99eb99c1 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Requested vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:14:26.275718 26817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8bd7225f5c48feb980049b99eb99c1" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:14:26.275945 26817 raft_consensus.cc:3037] T 8a8bd7225f5c48feb980049b99eb99c1 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:26.275916 26557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8bd7225f5c48feb980049b99eb99c1" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:14:26.276121 26557 raft_consensus.cc:3037] T 8a8bd7225f5c48feb980049b99eb99c1 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:26.276048 26280 catalog_manager.cc:5484] T a7aae43aeafc491b84d879fecfbd983f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195). New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:14:26.279045 26817 raft_consensus.cc:2454] T 8a8bd7225f5c48feb980049b99eb99c1 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:14:26.279045 26557 raft_consensus.cc:2454] T 8a8bd7225f5c48feb980049b99eb99c1 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:14:26.279665 26362 leader_election.cc:302] T 8a8bd7225f5c48feb980049b99eb99c1 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:26.280234 27781 raft_consensus.cc:2781] T 8a8bd7225f5c48feb980049b99eb99c1 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:26.280539 27781 raft_consensus.cc:686] T 8a8bd7225f5c48feb980049b99eb99c1 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Becoming Leader. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Running, Role: LEADER
I20240502 14:14:26.280843 27781 consensus_queue.cc:237] T 8a8bd7225f5c48feb980049b99eb99c1 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:26.283700 26280 catalog_manager.cc:5484] T 8a8bd7225f5c48feb980049b99eb99c1 P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: term changed from 0 to 1, leader changed from <none> to 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193). New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
W20240502 14:14:26.636497 27787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:26.636864 27787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 14:14:26.641935 27781 consensus_queue.cc:1035] T b8dd27ac333542a4844172fdc8e9b9bd P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:26.646752 27781 consensus_queue.cc:1035] T b8dd27ac333542a4844172fdc8e9b9bd P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:26.648273 27787 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:14:26.685755 27783 consensus_queue.cc:1035] T a7aae43aeafc491b84d879fecfbd983f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_4'
I20240502 14:14:26.694818 27781 consensus_queue.cc:1035] T 8a8bd7225f5c48feb980049b99eb99c1 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:26.695510 27783 consensus_queue.cc:1035] T a7aae43aeafc491b84d879fecfbd983f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:26.709007 27575 consensus_queue.cc:1035] T 8a8bd7225f5c48feb980049b99eb99c1 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 219.598 ms
time per row: 0.109799 ms
I20240502 14:14:27.075572 26280 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:27.085440 26667 tablet_service.cc:1467] Processing CreateTablet for tablet c9fb1f7eb71c42d2b8f0f6aacd2b0aeb (DEFAULT_TABLE table=rebalance_test_table_5 [id=8069f2786bf84bf4a012c26e82757ad4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:27.085440 26797 tablet_service.cc:1467] Processing CreateTablet for tablet c9fb1f7eb71c42d2b8f0f6aacd2b0aeb (DEFAULT_TABLE table=rebalance_test_table_5 [id=8069f2786bf84bf4a012c26e82757ad4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:27.085736 26537 tablet_service.cc:1467] Processing CreateTablet for tablet c9fb1f7eb71c42d2b8f0f6aacd2b0aeb (DEFAULT_TABLE table=rebalance_test_table_5 [id=8069f2786bf84bf4a012c26e82757ad4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:27.085932 26667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9fb1f7eb71c42d2b8f0f6aacd2b0aeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:27.085932 26797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9fb1f7eb71c42d2b8f0f6aacd2b0aeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:27.086201 26537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9fb1f7eb71c42d2b8f0f6aacd2b0aeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:27.086609 26536 tablet_service.cc:1467] Processing CreateTablet for tablet 4bf6e7410afa4c048f5842ec2a426d77 (DEFAULT_TABLE table=rebalance_test_table_5 [id=8069f2786bf84bf4a012c26e82757ad4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:27.086993 26536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bf6e7410afa4c048f5842ec2a426d77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:27.089282 26799 tablet_service.cc:1467] Processing CreateTablet for tablet 4bf6e7410afa4c048f5842ec2a426d77 (DEFAULT_TABLE table=rebalance_test_table_5 [id=8069f2786bf84bf4a012c26e82757ad4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:27.089355 26668 tablet_service.cc:1467] Processing CreateTablet for tablet 4bf6e7410afa4c048f5842ec2a426d77 (DEFAULT_TABLE table=rebalance_test_table_5 [id=8069f2786bf84bf4a012c26e82757ad4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:27.089726 26799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bf6e7410afa4c048f5842ec2a426d77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:27.089779 26668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bf6e7410afa4c048f5842ec2a426d77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:27.099812 26407 tablet_service.cc:1467] Processing CreateTablet for tablet 226b35583ee4441a82a66520aeffbb7d (DEFAULT_TABLE table=rebalance_test_table_5 [id=8069f2786bf84bf4a012c26e82757ad4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:27.100327 26407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 226b35583ee4441a82a66520aeffbb7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:27.100487 26668 tablet_service.cc:1467] Processing CreateTablet for tablet 226b35583ee4441a82a66520aeffbb7d (DEFAULT_TABLE table=rebalance_test_table_5 [id=8069f2786bf84bf4a012c26e82757ad4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:27.101064 26668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 226b35583ee4441a82a66520aeffbb7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:27.103718 27853 tablet_bootstrap.cc:492] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:27.105926 27853 tablet_bootstrap.cc:654] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:27.107934 27854 tablet_bootstrap.cc:492] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:27.108156 26537 tablet_service.cc:1467] Processing CreateTablet for tablet 226b35583ee4441a82a66520aeffbb7d (DEFAULT_TABLE table=rebalance_test_table_5 [id=8069f2786bf84bf4a012c26e82757ad4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:27.108798 26537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 226b35583ee4441a82a66520aeffbb7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:27.113124 27854 tablet_bootstrap.cc:654] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:27.115944 27852 tablet_bootstrap.cc:492] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:27.116876 27856 tablet_bootstrap.cc:492] T 226b35583ee4441a82a66520aeffbb7d P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:27.117974 27852 tablet_bootstrap.cc:654] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:27.119318 27856 tablet_bootstrap.cc:654] T 226b35583ee4441a82a66520aeffbb7d P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:27.121296 27853 tablet_bootstrap.cc:492] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:27.121503 27853 ts_tablet_manager.cc:1367] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.009s
I20240502 14:14:27.122437 27853 raft_consensus.cc:348] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:27.122800 27853 raft_consensus.cc:374] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:27.122984 27854 tablet_bootstrap.cc:492] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:27.122963 27853 raft_consensus.cc:729] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:27.123185 27854 ts_tablet_manager.cc:1367] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20240502 14:14:27.123373 27853 consensus_queue.cc:260] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:27.124146 27853 ts_tablet_manager.cc:1398] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:14:27.124898 27854 raft_consensus.cc:348] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:27.125128 27852 tablet_bootstrap.cc:492] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:27.125136 27854 raft_consensus.cc:374] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:27.125243 27854 raft_consensus.cc:729] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:27.125373 27852 ts_tablet_manager.cc:1367] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.001s
I20240502 14:14:27.125522 27854 consensus_queue.cc:260] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:27.126152 27854 ts_tablet_manager.cc:1398] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:27.126468 27854 tablet_bootstrap.cc:492] T 4bf6e7410afa4c048f5842ec2a426d77 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:27.126509 27852 raft_consensus.cc:348] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:27.126838 27852 raft_consensus.cc:374] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:27.126967 27852 raft_consensus.cc:729] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:27.127316 27852 consensus_queue.cc:260] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:27.129016 27854 tablet_bootstrap.cc:654] T 4bf6e7410afa4c048f5842ec2a426d77 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:27.129186 27852 ts_tablet_manager.cc:1398] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 14:14:27.129619 27852 tablet_bootstrap.cc:492] T 4bf6e7410afa4c048f5842ec2a426d77 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:27.131852 27852 tablet_bootstrap.cc:654] T 4bf6e7410afa4c048f5842ec2a426d77 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:27.134428 27856 tablet_bootstrap.cc:492] T 226b35583ee4441a82a66520aeffbb7d P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:27.134725 27856 ts_tablet_manager.cc:1367] T 226b35583ee4441a82a66520aeffbb7d P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.016s
I20240502 14:14:27.135802 27442 raft_consensus.cc:482] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:27.136052 27852 tablet_bootstrap.cc:492] T 4bf6e7410afa4c048f5842ec2a426d77 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:27.135967 27442 raft_consensus.cc:504] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:27.136206 27852 ts_tablet_manager.cc:1367] T 4bf6e7410afa4c048f5842ec2a426d77 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20240502 14:14:27.136806 27442 leader_election.cc:288] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:14:27.137102 27852 raft_consensus.cc:348] T 4bf6e7410afa4c048f5842ec2a426d77 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:27.137346 27852 raft_consensus.cc:374] T 4bf6e7410afa4c048f5842ec2a426d77 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:27.137432 27852 raft_consensus.cc:729] T 4bf6e7410afa4c048f5842ec2a426d77 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:27.137827 26687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fb1f7eb71c42d2b8f0f6aacd2b0aeb" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:14:27.138134 26687 raft_consensus.cc:2454] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 0.
I20240502 14:14:27.138288 27856 raft_consensus.cc:348] T 226b35583ee4441a82a66520aeffbb7d P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:27.138629 27856 raft_consensus.cc:374] T 226b35583ee4441a82a66520aeffbb7d P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:27.138657 26755 leader_election.cc:302] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:27.138842 27856 raft_consensus.cc:729] T 226b35583ee4441a82a66520aeffbb7d P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:27.138978 27442 raft_consensus.cc:2781] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:27.139096 27442 raft_consensus.cc:482] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:27.139217 27442 raft_consensus.cc:3037] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:27.139513 27856 consensus_queue.cc:260] T 226b35583ee4441a82a66520aeffbb7d P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:27.139330 27580 raft_consensus.cc:482] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:27.138126 26557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fb1f7eb71c42d2b8f0f6aacd2b0aeb" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:27.140058 27580 raft_consensus.cc:504] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:27.124522 27853 tablet_bootstrap.cc:492] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
W20240502 14:14:27.140861 26754 leader_election.cc:341] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:27.141685 27580 leader_election.cc:288] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:27.142204 27442 raft_consensus.cc:504] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:27.142874 26817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bf6e7410afa4c048f5842ec2a426d77" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:14:27.143419 26689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fb1f7eb71c42d2b8f0f6aacd2b0aeb" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:14:27.143604 26689 raft_consensus.cc:3037] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:27.144680 27856 ts_tablet_manager.cc:1398] T 226b35583ee4441a82a66520aeffbb7d P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
W20240502 14:14:27.145213 26494 leader_election.cc:341] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:27.146090 27853 tablet_bootstrap.cc:654] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:27.146237 26689 raft_consensus.cc:2454] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
I20240502 14:14:27.143370 26687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bf6e7410afa4c048f5842ec2a426d77" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:14:27.146903 26755 leader_election.cc:302] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:27.148646 26557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fb1f7eb71c42d2b8f0f6aacd2b0aeb" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:14:27.149210 27442 leader_election.cc:288] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Requested vote from peers 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
W20240502 14:14:27.149272 26754 leader_election.cc:341] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:27.149447 27442 raft_consensus.cc:2781] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:27.149586 27442 raft_consensus.cc:686] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Becoming Leader. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Running, Role: LEADER
I20240502 14:14:27.149876 27442 consensus_queue.cc:237] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:27.152705 27852 consensus_queue.cc:260] T 4bf6e7410afa4c048f5842ec2a426d77 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:27.153273 26687 raft_consensus.cc:2454] T 4bf6e7410afa4c048f5842ec2a426d77 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 0.
I20240502 14:14:27.153961 27852 ts_tablet_manager.cc:1398] T 4bf6e7410afa4c048f5842ec2a426d77 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.018s user 0.000s sys 0.002s
I20240502 14:14:27.154352 27852 tablet_bootstrap.cc:492] T 226b35583ee4441a82a66520aeffbb7d P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:27.154273 26494 leader_election.cc:302] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, ba187e38a2754d60a12de9ac4856430a; no voters: e4cfb8fc5a474177af6848a6efa13a27
I20240502 14:14:27.154799 27580 raft_consensus.cc:2781] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:27.154911 27580 raft_consensus.cc:482] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:27.155031 27580 raft_consensus.cc:3037] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:27.156440 27852 tablet_bootstrap.cc:654] T 226b35583ee4441a82a66520aeffbb7d P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:27.157307 26280 catalog_manager.cc:5484] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196). New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:14:27.158769 27854 tablet_bootstrap.cc:492] T 4bf6e7410afa4c048f5842ec2a426d77 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:27.158787 27580 raft_consensus.cc:504] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:27.159006 27854 ts_tablet_manager.cc:1367] T 4bf6e7410afa4c048f5842ec2a426d77 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.025s
I20240502 14:14:27.159911 27854 raft_consensus.cc:348] T 4bf6e7410afa4c048f5842ec2a426d77 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:27.159991 26687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bf6e7410afa4c048f5842ec2a426d77" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:14:27.160168 27854 raft_consensus.cc:374] T 4bf6e7410afa4c048f5842ec2a426d77 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:27.160214 26687 raft_consensus.cc:3037] T 4bf6e7410afa4c048f5842ec2a426d77 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:27.160142 26817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bf6e7410afa4c048f5842ec2a426d77" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:14:27.160264 27854 raft_consensus.cc:729] T 4bf6e7410afa4c048f5842ec2a426d77 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:27.160578 27854 consensus_queue.cc:260] T 4bf6e7410afa4c048f5842ec2a426d77 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:27.160845 26817 raft_consensus.cc:3037] T 4bf6e7410afa4c048f5842ec2a426d77 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:27.162742 26687 raft_consensus.cc:2454] T 4bf6e7410afa4c048f5842ec2a426d77 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:14:27.163125 27580 leader_election.cc:288] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Requested vote from peers 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:27.163267 26494 leader_election.cc:302] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:27.163756 27580 raft_consensus.cc:2781] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:27.164090 27580 raft_consensus.cc:686] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Becoming Leader. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Running, Role: LEADER
I20240502 14:14:27.164378 27580 consensus_queue.cc:237] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:27.164645 26817 raft_consensus.cc:2454] T 4bf6e7410afa4c048f5842ec2a426d77 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:14:27.165266 27854 ts_tablet_manager.cc:1398] T 4bf6e7410afa4c048f5842ec2a426d77 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 14:14:27.166359 27853 tablet_bootstrap.cc:492] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:27.166546 27853 ts_tablet_manager.cc:1367] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.042s user 0.007s sys 0.001s
I20240502 14:14:27.167192 27852 tablet_bootstrap.cc:492] T 226b35583ee4441a82a66520aeffbb7d P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:27.167423 27852 ts_tablet_manager.cc:1367] T 226b35583ee4441a82a66520aeffbb7d P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.004s
I20240502 14:14:27.167480 27853 raft_consensus.cc:348] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:27.167727 27853 raft_consensus.cc:374] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:27.167819 27853 raft_consensus.cc:729] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:27.168056 27853 consensus_queue.cc:260] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:27.168695 27853 ts_tablet_manager.cc:1398] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240502 14:14:27.168884 27852 raft_consensus.cc:348] T 226b35583ee4441a82a66520aeffbb7d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:27.169052 27853 tablet_bootstrap.cc:492] T 226b35583ee4441a82a66520aeffbb7d P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:27.169142 27852 raft_consensus.cc:374] T 226b35583ee4441a82a66520aeffbb7d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:27.169263 27852 raft_consensus.cc:729] T 226b35583ee4441a82a66520aeffbb7d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:27.169497 27852 consensus_queue.cc:260] T 226b35583ee4441a82a66520aeffbb7d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:27.170133 27852 ts_tablet_manager.cc:1398] T 226b35583ee4441a82a66520aeffbb7d P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:27.170840 27853 tablet_bootstrap.cc:654] T 226b35583ee4441a82a66520aeffbb7d P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:27.174005 26280 catalog_manager.cc:5484] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a reported cstate change: term changed from 0 to 1, leader changed from <none> to ba187e38a2754d60a12de9ac4856430a (127.19.94.194). New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:14:27.175288 27837 raft_consensus.cc:482] T 226b35583ee4441a82a66520aeffbb7d P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:27.175449 27837 raft_consensus.cc:504] T 226b35583ee4441a82a66520aeffbb7d P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:27.176218 27837 leader_election.cc:288] T 226b35583ee4441a82a66520aeffbb7d P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:14:27.176721 26557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "226b35583ee4441a82a66520aeffbb7d" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:27.176873 26687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "226b35583ee4441a82a66520aeffbb7d" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:14:27.177088 26687 raft_consensus.cc:2454] T 226b35583ee4441a82a66520aeffbb7d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:14:27.177515 26363 leader_election.cc:302] T 226b35583ee4441a82a66520aeffbb7d P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:14:27.177773 27837 raft_consensus.cc:2781] T 226b35583ee4441a82a66520aeffbb7d P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:27.177877 27837 raft_consensus.cc:482] T 226b35583ee4441a82a66520aeffbb7d P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:27.177971 27837 raft_consensus.cc:3037] T 226b35583ee4441a82a66520aeffbb7d P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:27.189055 27853 tablet_bootstrap.cc:492] T 226b35583ee4441a82a66520aeffbb7d P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:27.189273 27853 ts_tablet_manager.cc:1367] T 226b35583ee4441a82a66520aeffbb7d P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20240502 14:14:27.189949 27853 raft_consensus.cc:348] T 226b35583ee4441a82a66520aeffbb7d P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:27.190130 27853 raft_consensus.cc:374] T 226b35583ee4441a82a66520aeffbb7d P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:27.190202 27853 raft_consensus.cc:729] T 226b35583ee4441a82a66520aeffbb7d P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:27.190397 27853 consensus_queue.cc:260] T 226b35583ee4441a82a66520aeffbb7d P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:27.190812 27853 ts_tablet_manager.cc:1398] T 226b35583ee4441a82a66520aeffbb7d P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240502 14:14:27.200672 26362 leader_election.cc:341] T 226b35583ee4441a82a66520aeffbb7d P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:27.203392 27837 raft_consensus.cc:504] T 226b35583ee4441a82a66520aeffbb7d P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:27.204358 27837 leader_election.cc:288] T 226b35583ee4441a82a66520aeffbb7d P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Requested vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:14:27.204792 26557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "226b35583ee4441a82a66520aeffbb7d" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:14:27.205029 26557 raft_consensus.cc:3037] T 226b35583ee4441a82a66520aeffbb7d P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:27.204981 26687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "226b35583ee4441a82a66520aeffbb7d" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:14:27.205191 26687 raft_consensus.cc:3037] T 226b35583ee4441a82a66520aeffbb7d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:27.208114 26557 raft_consensus.cc:2454] T 226b35583ee4441a82a66520aeffbb7d P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:14:27.208124 26687 raft_consensus.cc:2454] T 226b35583ee4441a82a66520aeffbb7d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:14:27.208626 26362 leader_election.cc:302] T 226b35583ee4441a82a66520aeffbb7d P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:27.209064 27837 raft_consensus.cc:2781] T 226b35583ee4441a82a66520aeffbb7d P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:27.209445 27837 raft_consensus.cc:686] T 226b35583ee4441a82a66520aeffbb7d P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Becoming Leader. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Running, Role: LEADER
I20240502 14:14:27.209826 27837 consensus_queue.cc:237] T 226b35583ee4441a82a66520aeffbb7d P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:27.213227 26280 catalog_manager.cc:5484] T 226b35583ee4441a82a66520aeffbb7d P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: term changed from 0 to 1, leader changed from <none> to 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193). New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:14:27.424443 27580 raft_consensus.cc:482] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:27.424823 27580 raft_consensus.cc:504] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:27.425643 27580 leader_election.cc:288] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:27.426411 26687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fb1f7eb71c42d2b8f0f6aacd2b0aeb" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:14:27.426482 26817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fb1f7eb71c42d2b8f0f6aacd2b0aeb" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:14:27.426769 26687 raft_consensus.cc:2379] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ba187e38a2754d60a12de9ac4856430a in current term 1: Already voted for candidate e4cfb8fc5a474177af6848a6efa13a27 in this term.
I20240502 14:14:27.427287 26494 leader_election.cc:302] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, e4cfb8fc5a474177af6848a6efa13a27
I20240502 14:14:27.427582 27580 raft_consensus.cc:3037] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:27.429708 27580 raft_consensus.cc:2726] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:14:27.584151 27580 consensus_queue.cc:1035] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:27.591846 27580 consensus_queue.cc:1035] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:27.595657 27869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:27.596261 27869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:27.605577 27869 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240502 14:14:27.667425 27863 consensus_queue.cc:1035] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:27.674872 27863 consensus_queue.cc:1035] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:27.768987 27837 consensus_queue.cc:1035] T 226b35583ee4441a82a66520aeffbb7d P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:27.820345 27920 consensus_queue.cc:1035] T 226b35583ee4441a82a66520aeffbb7d P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 287.319 ms
time per row: 0.14366 ms
I20240502 14:14:28.069193 26282 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:28.078979 26407 tablet_service.cc:1467] Processing CreateTablet for tablet df0c1182765041a7ad4f28ef6aff35f0 (DEFAULT_TABLE table=rebalance_test_table_6 [id=78d3d1a6e46640a0a44b6b531584cbe1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:28.079205 26537 tablet_service.cc:1467] Processing CreateTablet for tablet df0c1182765041a7ad4f28ef6aff35f0 (DEFAULT_TABLE table=rebalance_test_table_6 [id=78d3d1a6e46640a0a44b6b531584cbe1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:28.079521 26407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df0c1182765041a7ad4f28ef6aff35f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:28.079620 26537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df0c1182765041a7ad4f28ef6aff35f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:28.080760 26536 tablet_service.cc:1467] Processing CreateTablet for tablet 78954e9f7e3147e288eb0cbf6396d345 (DEFAULT_TABLE table=rebalance_test_table_6 [id=78d3d1a6e46640a0a44b6b531584cbe1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:28.081259 26536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78954e9f7e3147e288eb0cbf6396d345. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:28.082537 26798 tablet_service.cc:1467] Processing CreateTablet for tablet 78954e9f7e3147e288eb0cbf6396d345 (DEFAULT_TABLE table=rebalance_test_table_6 [id=78d3d1a6e46640a0a44b6b531584cbe1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:28.082770 26799 tablet_service.cc:1467] Processing CreateTablet for tablet 6d8b293ae36740d08d64bdeeb197897d (DEFAULT_TABLE table=rebalance_test_table_6 [id=78d3d1a6e46640a0a44b6b531584cbe1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:28.083005 26798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78954e9f7e3147e288eb0cbf6396d345. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:28.083822 26799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d8b293ae36740d08d64bdeeb197897d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:28.084075 26406 tablet_service.cc:1467] Processing CreateTablet for tablet 78954e9f7e3147e288eb0cbf6396d345 (DEFAULT_TABLE table=rebalance_test_table_6 [id=78d3d1a6e46640a0a44b6b531584cbe1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:28.084702 26406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78954e9f7e3147e288eb0cbf6396d345. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:28.089097 26668 tablet_service.cc:1467] Processing CreateTablet for tablet 6d8b293ae36740d08d64bdeeb197897d (DEFAULT_TABLE table=rebalance_test_table_6 [id=78d3d1a6e46640a0a44b6b531584cbe1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:28.089735 26668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d8b293ae36740d08d64bdeeb197897d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:28.090991 27936 tablet_bootstrap.cc:492] T df0c1182765041a7ad4f28ef6aff35f0 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:28.091392 26405 tablet_service.cc:1467] Processing CreateTablet for tablet 6d8b293ae36740d08d64bdeeb197897d (DEFAULT_TABLE table=rebalance_test_table_6 [id=78d3d1a6e46640a0a44b6b531584cbe1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:28.091887 26405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d8b293ae36740d08d64bdeeb197897d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:28.093159 27936 tablet_bootstrap.cc:654] T df0c1182765041a7ad4f28ef6aff35f0 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:28.097177 26797 tablet_service.cc:1467] Processing CreateTablet for tablet df0c1182765041a7ad4f28ef6aff35f0 (DEFAULT_TABLE table=rebalance_test_table_6 [id=78d3d1a6e46640a0a44b6b531584cbe1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:28.098040 26797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df0c1182765041a7ad4f28ef6aff35f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:28.103950 27939 tablet_bootstrap.cc:492] T 78954e9f7e3147e288eb0cbf6396d345 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:28.107895 27939 tablet_bootstrap.cc:654] T 78954e9f7e3147e288eb0cbf6396d345 P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:28.109885 27937 tablet_bootstrap.cc:492] T 6d8b293ae36740d08d64bdeeb197897d P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:28.112139 27937 tablet_bootstrap.cc:654] T 6d8b293ae36740d08d64bdeeb197897d P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:28.113242 27939 tablet_bootstrap.cc:492] T 78954e9f7e3147e288eb0cbf6396d345 P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:28.113514 27939 ts_tablet_manager.cc:1367] T 78954e9f7e3147e288eb0cbf6396d345 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20240502 14:14:28.114709 27939 raft_consensus.cc:348] T 78954e9f7e3147e288eb0cbf6396d345 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:28.115093 27939 raft_consensus.cc:374] T 78954e9f7e3147e288eb0cbf6396d345 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:28.115270 27939 raft_consensus.cc:729] T 78954e9f7e3147e288eb0cbf6396d345 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:28.115638 27939 consensus_queue.cc:260] T 78954e9f7e3147e288eb0cbf6396d345 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:28.116472 27939 ts_tablet_manager.cc:1398] T 78954e9f7e3147e288eb0cbf6396d345 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 14:14:28.130930 27939 tablet_bootstrap.cc:492] T df0c1182765041a7ad4f28ef6aff35f0 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:28.133708 27939 tablet_bootstrap.cc:654] T df0c1182765041a7ad4f28ef6aff35f0 P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:28.137380 27939 tablet_bootstrap.cc:492] T df0c1182765041a7ad4f28ef6aff35f0 P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:28.137917 27939 ts_tablet_manager.cc:1367] T df0c1182765041a7ad4f28ef6aff35f0 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240502 14:14:28.138886 27943 tablet_bootstrap.cc:492] T 6d8b293ae36740d08d64bdeeb197897d P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:28.139798 27939 raft_consensus.cc:348] T df0c1182765041a7ad4f28ef6aff35f0 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:28.140177 27936 tablet_bootstrap.cc:492] T df0c1182765041a7ad4f28ef6aff35f0 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:28.140350 27936 ts_tablet_manager.cc:1367] T df0c1182765041a7ad4f28ef6aff35f0 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.049s user 0.023s sys 0.024s
I20240502 14:14:28.140429 27937 tablet_bootstrap.cc:492] T 6d8b293ae36740d08d64bdeeb197897d P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:28.140619 27937 ts_tablet_manager.cc:1367] T 6d8b293ae36740d08d64bdeeb197897d P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.023s
I20240502 14:14:28.141108 27936 raft_consensus.cc:348] T df0c1182765041a7ad4f28ef6aff35f0 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:28.141292 27936 raft_consensus.cc:374] T df0c1182765041a7ad4f28ef6aff35f0 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:28.141425 27936 raft_consensus.cc:729] T df0c1182765041a7ad4f28ef6aff35f0 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:28.141458 27937 raft_consensus.cc:348] T 6d8b293ae36740d08d64bdeeb197897d P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:28.141697 27937 raft_consensus.cc:374] T 6d8b293ae36740d08d64bdeeb197897d P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:28.141817 27937 raft_consensus.cc:729] T 6d8b293ae36740d08d64bdeeb197897d P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:28.141813 27939 raft_consensus.cc:374] T df0c1182765041a7ad4f28ef6aff35f0 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:28.141698 27936 consensus_queue.cc:260] T df0c1182765041a7ad4f28ef6aff35f0 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:28.141908 27939 raft_consensus.cc:729] T df0c1182765041a7ad4f28ef6aff35f0 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:28.142115 27937 consensus_queue.cc:260] T 6d8b293ae36740d08d64bdeeb197897d P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:28.142335 27936 ts_tablet_manager.cc:1398] T df0c1182765041a7ad4f28ef6aff35f0 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:14:28.142524 27939 consensus_queue.cc:260] T df0c1182765041a7ad4f28ef6aff35f0 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:28.142943 27937 ts_tablet_manager.cc:1398] T 6d8b293ae36740d08d64bdeeb197897d P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:28.143234 27937 tablet_bootstrap.cc:492] T 78954e9f7e3147e288eb0cbf6396d345 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:28.145017 27936 tablet_bootstrap.cc:492] T 78954e9f7e3147e288eb0cbf6396d345 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:28.145076 27937 tablet_bootstrap.cc:654] T 78954e9f7e3147e288eb0cbf6396d345 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:28.145104 27939 ts_tablet_manager.cc:1398] T df0c1182765041a7ad4f28ef6aff35f0 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:14:28.146867 27943 tablet_bootstrap.cc:654] T 6d8b293ae36740d08d64bdeeb197897d P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:28.149591 27936 tablet_bootstrap.cc:654] T 78954e9f7e3147e288eb0cbf6396d345 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:28.153317 27920 raft_consensus.cc:482] T df0c1182765041a7ad4f28ef6aff35f0 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:28.153539 27920 raft_consensus.cc:504] T df0c1182765041a7ad4f28ef6aff35f0 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:28.153909 27937 tablet_bootstrap.cc:492] T 78954e9f7e3147e288eb0cbf6396d345 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:28.153973 27936 tablet_bootstrap.cc:492] T 78954e9f7e3147e288eb0cbf6396d345 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:28.154021 27943 tablet_bootstrap.cc:492] T 6d8b293ae36740d08d64bdeeb197897d P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:28.154111 27937 ts_tablet_manager.cc:1367] T 78954e9f7e3147e288eb0cbf6396d345 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.001s
I20240502 14:14:28.154167 27943 ts_tablet_manager.cc:1367] T 6d8b293ae36740d08d64bdeeb197897d P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.004s
I20240502 14:14:28.154151 27936 ts_tablet_manager.cc:1367] T 78954e9f7e3147e288eb0cbf6396d345 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.001s
I20240502 14:14:28.154306 27920 leader_election.cc:288] T df0c1182765041a7ad4f28ef6aff35f0 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:28.154973 27943 raft_consensus.cc:348] T 6d8b293ae36740d08d64bdeeb197897d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:28.154964 27937 raft_consensus.cc:348] T 78954e9f7e3147e288eb0cbf6396d345 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:28.154999 27936 raft_consensus.cc:348] T 78954e9f7e3147e288eb0cbf6396d345 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:28.155226 27937 raft_consensus.cc:374] T 78954e9f7e3147e288eb0cbf6396d345 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:28.155232 27936 raft_consensus.cc:374] T 78954e9f7e3147e288eb0cbf6396d345 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:28.155315 27936 raft_consensus.cc:729] T 78954e9f7e3147e288eb0cbf6396d345 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:28.155262 26817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df0c1182765041a7ad4f28ef6aff35f0" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:14:28.155339 27943 raft_consensus.cc:374] T 6d8b293ae36740d08d64bdeeb197897d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:28.155355 27937 raft_consensus.cc:729] T 78954e9f7e3147e288eb0cbf6396d345 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:28.155504 27943 raft_consensus.cc:729] T 6d8b293ae36740d08d64bdeeb197897d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:28.155632 27936 consensus_queue.cc:260] T 78954e9f7e3147e288eb0cbf6396d345 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:28.155666 27937 consensus_queue.cc:260] T 78954e9f7e3147e288eb0cbf6396d345 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:28.156262 27936 ts_tablet_manager.cc:1398] T 78954e9f7e3147e288eb0cbf6396d345 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:14:28.156174 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df0c1182765041a7ad4f28ef6aff35f0" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:28.156474 26558 raft_consensus.cc:2454] T df0c1182765041a7ad4f28ef6aff35f0 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:14:28.156037 27943 consensus_queue.cc:260] T 6d8b293ae36740d08d64bdeeb197897d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:28.156666 27936 tablet_bootstrap.cc:492] T 6d8b293ae36740d08d64bdeeb197897d P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:28.157187 27943 ts_tablet_manager.cc:1398] T 6d8b293ae36740d08d64bdeeb197897d P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:14:28.157886 27937 ts_tablet_manager.cc:1398] T 78954e9f7e3147e288eb0cbf6396d345 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
W20240502 14:14:28.157955 26363 leader_election.cc:341] T df0c1182765041a7ad4f28ef6aff35f0 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:28.158291 27937 tablet_bootstrap.cc:492] T df0c1182765041a7ad4f28ef6aff35f0 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:28.158471 26362 leader_election.cc:302] T df0c1182765041a7ad4f28ef6aff35f0 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, ba187e38a2754d60a12de9ac4856430a; no voters: e4cfb8fc5a474177af6848a6efa13a27
I20240502 14:14:28.159019 27837 raft_consensus.cc:2781] T df0c1182765041a7ad4f28ef6aff35f0 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:28.159127 27837 raft_consensus.cc:482] T df0c1182765041a7ad4f28ef6aff35f0 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:28.159268 27837 raft_consensus.cc:3037] T df0c1182765041a7ad4f28ef6aff35f0 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:28.161063 27936 tablet_bootstrap.cc:654] T 6d8b293ae36740d08d64bdeeb197897d P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:28.161443 27937 tablet_bootstrap.cc:654] T df0c1182765041a7ad4f28ef6aff35f0 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:28.162520 27837 raft_consensus.cc:504] T df0c1182765041a7ad4f28ef6aff35f0 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:28.164429 26817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df0c1182765041a7ad4f28ef6aff35f0" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
W20240502 14:14:28.165172 26363 leader_election.cc:341] T df0c1182765041a7ad4f28ef6aff35f0 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:28.167575 27836 raft_consensus.cc:482] T 78954e9f7e3147e288eb0cbf6396d345 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:28.167781 27836 raft_consensus.cc:504] T 78954e9f7e3147e288eb0cbf6396d345 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:28.168726 27836 leader_election.cc:288] T 78954e9f7e3147e288eb0cbf6396d345 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:14:28.169857 26557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df0c1182765041a7ad4f28ef6aff35f0" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:14:28.170111 26557 raft_consensus.cc:3037] T df0c1182765041a7ad4f28ef6aff35f0 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:28.170437 27936 tablet_bootstrap.cc:492] T 6d8b293ae36740d08d64bdeeb197897d P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:28.170450 27937 tablet_bootstrap.cc:492] T df0c1182765041a7ad4f28ef6aff35f0 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:28.170589 27936 ts_tablet_manager.cc:1367] T 6d8b293ae36740d08d64bdeeb197897d P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20240502 14:14:28.170603 27937 ts_tablet_manager.cc:1367] T df0c1182765041a7ad4f28ef6aff35f0 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.009s
I20240502 14:14:28.171200 26817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78954e9f7e3147e288eb0cbf6396d345" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:14:28.171424 26817 raft_consensus.cc:2454] T 78954e9f7e3147e288eb0cbf6396d345 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:14:28.171442 27936 raft_consensus.cc:348] T 6d8b293ae36740d08d64bdeeb197897d P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:28.171453 27937 raft_consensus.cc:348] T df0c1182765041a7ad4f28ef6aff35f0 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:28.171648 27936 raft_consensus.cc:374] T 6d8b293ae36740d08d64bdeeb197897d P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:28.171679 27937 raft_consensus.cc:374] T df0c1182765041a7ad4f28ef6aff35f0 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:28.171746 27936 raft_consensus.cc:729] T 6d8b293ae36740d08d64bdeeb197897d P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:28.171772 27937 raft_consensus.cc:729] T df0c1182765041a7ad4f28ef6aff35f0 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:28.171972 27936 consensus_queue.cc:260] T 6d8b293ae36740d08d64bdeeb197897d P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:28.172008 27937 consensus_queue.cc:260] T df0c1182765041a7ad4f28ef6aff35f0 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:28.172248 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78954e9f7e3147e288eb0cbf6396d345" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:28.172497 26558 raft_consensus.cc:2454] T 78954e9f7e3147e288eb0cbf6396d345 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:14:28.172580 27936 ts_tablet_manager.cc:1398] T 6d8b293ae36740d08d64bdeeb197897d P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 14:14:28.172586 27937 ts_tablet_manager.cc:1398] T df0c1182765041a7ad4f28ef6aff35f0 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:14:28.173132 26557 raft_consensus.cc:2454] T df0c1182765041a7ad4f28ef6aff35f0 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:14:28.173954 26362 leader_election.cc:302] T 78954e9f7e3147e288eb0cbf6396d345 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:28.174254 27920 raft_consensus.cc:2781] T 78954e9f7e3147e288eb0cbf6396d345 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:28.174299 26362 leader_election.cc:302] T df0c1182765041a7ad4f28ef6aff35f0 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, ba187e38a2754d60a12de9ac4856430a; no voters: e4cfb8fc5a474177af6848a6efa13a27
I20240502 14:14:28.174366 27920 raft_consensus.cc:482] T 78954e9f7e3147e288eb0cbf6396d345 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:28.174512 27920 raft_consensus.cc:3037] T 78954e9f7e3147e288eb0cbf6396d345 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:28.175002 27836 raft_consensus.cc:2781] T df0c1182765041a7ad4f28ef6aff35f0 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:28.175639 27837 leader_election.cc:288] T df0c1182765041a7ad4f28ef6aff35f0 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Requested vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:28.177969 27920 raft_consensus.cc:504] T 78954e9f7e3147e288eb0cbf6396d345 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:28.179383 26817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78954e9f7e3147e288eb0cbf6396d345" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:14:28.179615 26817 raft_consensus.cc:3037] T 78954e9f7e3147e288eb0cbf6396d345 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:28.179623 26557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78954e9f7e3147e288eb0cbf6396d345" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:14:28.179831 26557 raft_consensus.cc:3037] T 78954e9f7e3147e288eb0cbf6396d345 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:28.182703 26557 raft_consensus.cc:2454] T 78954e9f7e3147e288eb0cbf6396d345 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:14:28.183171 26362 leader_election.cc:302] T 78954e9f7e3147e288eb0cbf6396d345 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:28.183449 27920 leader_election.cc:288] T 78954e9f7e3147e288eb0cbf6396d345 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Requested vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:14:28.183537 27837 raft_consensus.cc:2781] T 78954e9f7e3147e288eb0cbf6396d345 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:28.183779 27837 raft_consensus.cc:686] T 78954e9f7e3147e288eb0cbf6396d345 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Becoming Leader. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Running, Role: LEADER
I20240502 14:14:28.184069 26817 raft_consensus.cc:2454] T 78954e9f7e3147e288eb0cbf6396d345 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:14:28.184177 27837 consensus_queue.cc:237] T 78954e9f7e3147e288eb0cbf6396d345 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:28.189086 27836 raft_consensus.cc:686] T df0c1182765041a7ad4f28ef6aff35f0 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Becoming Leader. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Running, Role: LEADER
I20240502 14:14:28.189385 27836 consensus_queue.cc:237] T df0c1182765041a7ad4f28ef6aff35f0 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:28.192556 26280 catalog_manager.cc:5484] T 78954e9f7e3147e288eb0cbf6396d345 P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: term changed from 0 to 1, leader changed from <none> to 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193). New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:14:28.193007 26280 catalog_manager.cc:5484] T df0c1182765041a7ad4f28ef6aff35f0 P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: term changed from 0 to 1, leader changed from <none> to 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193). New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:14:28.366255 27783 raft_consensus.cc:482] T 6d8b293ae36740d08d64bdeeb197897d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:28.366482 27783 raft_consensus.cc:504] T 6d8b293ae36740d08d64bdeeb197897d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:28.367219 27783 leader_election.cc:288] T 6d8b293ae36740d08d64bdeeb197897d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:28.367782 26817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d8b293ae36740d08d64bdeeb197897d" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:14:28.367859 26427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d8b293ae36740d08d64bdeeb197897d" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:14:28.368027 26817 raft_consensus.cc:2454] T 6d8b293ae36740d08d64bdeeb197897d P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:14:28.368093 26427 raft_consensus.cc:2454] T 6d8b293ae36740d08d64bdeeb197897d P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:14:28.368485 26625 leader_election.cc:302] T 6d8b293ae36740d08d64bdeeb197897d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:28.368901 27783 raft_consensus.cc:2781] T 6d8b293ae36740d08d64bdeeb197897d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:28.369027 27783 raft_consensus.cc:482] T 6d8b293ae36740d08d64bdeeb197897d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:28.369127 27783 raft_consensus.cc:3037] T 6d8b293ae36740d08d64bdeeb197897d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:28.371693 27783 raft_consensus.cc:504] T 6d8b293ae36740d08d64bdeeb197897d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:28.372305 27783 leader_election.cc:288] T 6d8b293ae36740d08d64bdeeb197897d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Requested vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:28.372964 26427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d8b293ae36740d08d64bdeeb197897d" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:14:28.373243 26427 raft_consensus.cc:3037] T 6d8b293ae36740d08d64bdeeb197897d P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:28.375290 26427 raft_consensus.cc:2454] T 6d8b293ae36740d08d64bdeeb197897d P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:14:28.375680 26817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d8b293ae36740d08d64bdeeb197897d" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:14:28.375761 26624 leader_election.cc:302] T 6d8b293ae36740d08d64bdeeb197897d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:14:28.375941 26817 raft_consensus.cc:3037] T 6d8b293ae36740d08d64bdeeb197897d P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:28.376104 27783 raft_consensus.cc:2781] T 6d8b293ae36740d08d64bdeeb197897d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:28.376592 27783 raft_consensus.cc:686] T 6d8b293ae36740d08d64bdeeb197897d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Becoming Leader. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Running, Role: LEADER
I20240502 14:14:28.376920 27783 consensus_queue.cc:237] T 6d8b293ae36740d08d64bdeeb197897d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:28.378656 26817 raft_consensus.cc:2454] T 6d8b293ae36740d08d64bdeeb197897d P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:14:28.379992 26280 catalog_manager.cc:5484] T 6d8b293ae36740d08d64bdeeb197897d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195). New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:14:28.502105 27863 raft_consensus.cc:482] T df0c1182765041a7ad4f28ef6aff35f0 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:28.502334 27863 raft_consensus.cc:504] T df0c1182765041a7ad4f28ef6aff35f0 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:28.503065 27863 leader_election.cc:288] T df0c1182765041a7ad4f28ef6aff35f0 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:14:28.503749 26427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df0c1182765041a7ad4f28ef6aff35f0" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:14:28.503803 26557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df0c1182765041a7ad4f28ef6aff35f0" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:28.504069 26557 raft_consensus.cc:2379] T df0c1182765041a7ad4f28ef6aff35f0 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4cfb8fc5a474177af6848a6efa13a27 in current term 1: Already voted for candidate 89af7f423cf44717a08d6b142f0ed49d in this term.
I20240502 14:14:28.504575 26754 leader_election.cc:302] T df0c1182765041a7ad4f28ef6aff35f0 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4cfb8fc5a474177af6848a6efa13a27; no voters: 89af7f423cf44717a08d6b142f0ed49d, ba187e38a2754d60a12de9ac4856430a
I20240502 14:14:28.504886 27863 raft_consensus.cc:3037] T df0c1182765041a7ad4f28ef6aff35f0 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:28.506886 27863 raft_consensus.cc:2726] T df0c1182765041a7ad4f28ef6aff35f0 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:14:28.604521 27836 consensus_queue.cc:1035] T 78954e9f7e3147e288eb0cbf6396d345 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:28.611451 27949 consensus_queue.cc:1035] T 78954e9f7e3147e288eb0cbf6396d345 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:28.652033 27949 consensus_queue.cc:1035] T df0c1182765041a7ad4f28ef6aff35f0 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:28.656805 27949 consensus_queue.cc:1035] T df0c1182765041a7ad4f28ef6aff35f0 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:28.772218 27955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:28.772624 27955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:28.780642 27955 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:14:28.792757 27783 consensus_queue.cc:1035] T 6d8b293ae36740d08d64bdeeb197897d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:28.800151 27783 consensus_queue.cc:1035] T 6d8b293ae36740d08d64bdeeb197897d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_6'
W20240502 14:14:29.058526 26363 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.196:32843, user_credentials={real_user=slave}} blocked reactor thread for 57673.1us
INSERT report
rows total: 2000
time total: 256.705 ms
time per row: 0.128353 ms
I20240502 14:14:29.192675 26279 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:29.202422 26537 tablet_service.cc:1467] Processing CreateTablet for tablet 9f3cbedfa1da42698615a544d9dc3276 (DEFAULT_TABLE table=rebalance_test_table_7 [id=998b08830b5f4b6fbb49d494f21739b9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:29.202584 26668 tablet_service.cc:1467] Processing CreateTablet for tablet 9f3cbedfa1da42698615a544d9dc3276 (DEFAULT_TABLE table=rebalance_test_table_7 [id=998b08830b5f4b6fbb49d494f21739b9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:29.202972 26537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f3cbedfa1da42698615a544d9dc3276. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:29.203015 26668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f3cbedfa1da42698615a544d9dc3276. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:29.206017 26405 tablet_service.cc:1467] Processing CreateTablet for tablet 9f3cbedfa1da42698615a544d9dc3276 (DEFAULT_TABLE table=rebalance_test_table_7 [id=998b08830b5f4b6fbb49d494f21739b9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:29.206490 26405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f3cbedfa1da42698615a544d9dc3276. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:29.206605 26536 tablet_service.cc:1467] Processing CreateTablet for tablet d7f02f897ce54fefb03832da3c2d95e0 (DEFAULT_TABLE table=rebalance_test_table_7 [id=998b08830b5f4b6fbb49d494f21739b9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:29.207031 26536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7f02f897ce54fefb03832da3c2d95e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:29.208909 26798 tablet_service.cc:1467] Processing CreateTablet for tablet aedd72fd3aeb48d68f26a042069cebab (DEFAULT_TABLE table=rebalance_test_table_7 [id=998b08830b5f4b6fbb49d494f21739b9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:29.209337 26798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aedd72fd3aeb48d68f26a042069cebab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:29.212872 26667 tablet_service.cc:1467] Processing CreateTablet for tablet aedd72fd3aeb48d68f26a042069cebab (DEFAULT_TABLE table=rebalance_test_table_7 [id=998b08830b5f4b6fbb49d494f21739b9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:29.212949 26797 tablet_service.cc:1467] Processing CreateTablet for tablet d7f02f897ce54fefb03832da3c2d95e0 (DEFAULT_TABLE table=rebalance_test_table_7 [id=998b08830b5f4b6fbb49d494f21739b9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:29.213394 26667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aedd72fd3aeb48d68f26a042069cebab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:29.213449 26797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7f02f897ce54fefb03832da3c2d95e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:29.216924 26669 tablet_service.cc:1467] Processing CreateTablet for tablet d7f02f897ce54fefb03832da3c2d95e0 (DEFAULT_TABLE table=rebalance_test_table_7 [id=998b08830b5f4b6fbb49d494f21739b9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:29.217425 26669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7f02f897ce54fefb03832da3c2d95e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:29.205977 26406 tablet_service.cc:1467] Processing CreateTablet for tablet aedd72fd3aeb48d68f26a042069cebab (DEFAULT_TABLE table=rebalance_test_table_7 [id=998b08830b5f4b6fbb49d494f21739b9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:29.225535 26406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aedd72fd3aeb48d68f26a042069cebab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:29.228996 28015 tablet_bootstrap.cc:492] T 9f3cbedfa1da42698615a544d9dc3276 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:29.232900 28015 tablet_bootstrap.cc:654] T 9f3cbedfa1da42698615a544d9dc3276 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:29.234413 28016 tablet_bootstrap.cc:492] T aedd72fd3aeb48d68f26a042069cebab P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:29.237485 28014 tablet_bootstrap.cc:492] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:29.238811 28016 tablet_bootstrap.cc:654] T aedd72fd3aeb48d68f26a042069cebab P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:29.242115 28014 tablet_bootstrap.cc:654] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:29.247498 28014 tablet_bootstrap.cc:492] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:29.247511 28016 tablet_bootstrap.cc:492] T aedd72fd3aeb48d68f26a042069cebab P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:29.247704 28016 ts_tablet_manager.cc:1367] T aedd72fd3aeb48d68f26a042069cebab P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20240502 14:14:29.247756 28014 ts_tablet_manager.cc:1367] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20240502 14:14:29.248548 28014 raft_consensus.cc:348] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:29.248797 28014 raft_consensus.cc:374] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:29.248889 28014 raft_consensus.cc:729] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:29.250589 28016 raft_consensus.cc:348] T aedd72fd3aeb48d68f26a042069cebab P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:29.250977 28016 raft_consensus.cc:374] T aedd72fd3aeb48d68f26a042069cebab P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:29.249161 28014 consensus_queue.cc:260] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:29.251178 28016 raft_consensus.cc:729] T aedd72fd3aeb48d68f26a042069cebab P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:29.251524 28016 consensus_queue.cc:260] T aedd72fd3aeb48d68f26a042069cebab P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:29.252375 28014 ts_tablet_manager.cc:1398] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:14:29.253778 28016 ts_tablet_manager.cc:1398] T aedd72fd3aeb48d68f26a042069cebab P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:14:29.254400 28016 tablet_bootstrap.cc:492] T d7f02f897ce54fefb03832da3c2d95e0 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:29.257620 28014 tablet_bootstrap.cc:492] T aedd72fd3aeb48d68f26a042069cebab P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:29.258404 28016 tablet_bootstrap.cc:654] T d7f02f897ce54fefb03832da3c2d95e0 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:29.259383 28014 tablet_bootstrap.cc:654] T aedd72fd3aeb48d68f26a042069cebab P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:29.265918 28014 tablet_bootstrap.cc:492] T aedd72fd3aeb48d68f26a042069cebab P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:29.266105 28014 ts_tablet_manager.cc:1367] T aedd72fd3aeb48d68f26a042069cebab P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.002s
I20240502 14:14:29.267577 28016 tablet_bootstrap.cc:492] T d7f02f897ce54fefb03832da3c2d95e0 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:29.267874 28016 ts_tablet_manager.cc:1367] T d7f02f897ce54fefb03832da3c2d95e0 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20240502 14:14:29.268985 28014 raft_consensus.cc:348] T aedd72fd3aeb48d68f26a042069cebab P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:29.269238 28014 raft_consensus.cc:374] T aedd72fd3aeb48d68f26a042069cebab P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:29.269321 28014 raft_consensus.cc:729] T aedd72fd3aeb48d68f26a042069cebab P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:29.269042 28016 raft_consensus.cc:348] T d7f02f897ce54fefb03832da3c2d95e0 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:29.269681 28016 raft_consensus.cc:374] T d7f02f897ce54fefb03832da3c2d95e0 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:29.269642 28014 consensus_queue.cc:260] T aedd72fd3aeb48d68f26a042069cebab P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:29.269886 28016 raft_consensus.cc:729] T d7f02f897ce54fefb03832da3c2d95e0 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:29.270304 28016 consensus_queue.cc:260] T d7f02f897ce54fefb03832da3c2d95e0 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:29.272167 28016 ts_tablet_manager.cc:1398] T d7f02f897ce54fefb03832da3c2d95e0 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:14:29.277750 28014 ts_tablet_manager.cc:1398] T aedd72fd3aeb48d68f26a042069cebab P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20240502 14:14:29.284337 28013 tablet_bootstrap.cc:492] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:29.285818 28015 tablet_bootstrap.cc:492] T 9f3cbedfa1da42698615a544d9dc3276 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:29.286001 28015 ts_tablet_manager.cc:1367] T 9f3cbedfa1da42698615a544d9dc3276 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.057s user 0.014s sys 0.012s
I20240502 14:14:29.286808 28015 raft_consensus.cc:348] T 9f3cbedfa1da42698615a544d9dc3276 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:29.287034 28015 raft_consensus.cc:374] T 9f3cbedfa1da42698615a544d9dc3276 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:29.287156 28015 raft_consensus.cc:729] T 9f3cbedfa1da42698615a544d9dc3276 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:29.287369 28013 tablet_bootstrap.cc:654] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:29.287427 28015 consensus_queue.cc:260] T 9f3cbedfa1da42698615a544d9dc3276 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:29.288106 28015 ts_tablet_manager.cc:1398] T 9f3cbedfa1da42698615a544d9dc3276 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:14:29.288479 28015 tablet_bootstrap.cc:492] T aedd72fd3aeb48d68f26a042069cebab P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:29.290079 28015 tablet_bootstrap.cc:654] T aedd72fd3aeb48d68f26a042069cebab P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:29.294664 28013 tablet_bootstrap.cc:492] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:29.294837 28013 ts_tablet_manager.cc:1367] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20240502 14:14:29.295074 28015 tablet_bootstrap.cc:492] T aedd72fd3aeb48d68f26a042069cebab P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:29.295243 28015 ts_tablet_manager.cc:1367] T aedd72fd3aeb48d68f26a042069cebab P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20240502 14:14:29.295534 28013 raft_consensus.cc:348] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:29.295722 28013 raft_consensus.cc:374] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:29.295835 28013 raft_consensus.cc:729] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:29.295955 28015 raft_consensus.cc:348] T aedd72fd3aeb48d68f26a042069cebab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:29.296080 28013 consensus_queue.cc:260] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:29.296190 28015 raft_consensus.cc:374] T aedd72fd3aeb48d68f26a042069cebab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:29.296264 28015 raft_consensus.cc:729] T aedd72fd3aeb48d68f26a042069cebab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:29.296674 28013 ts_tablet_manager.cc:1398] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:14:29.296697 28015 consensus_queue.cc:260] T aedd72fd3aeb48d68f26a042069cebab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:29.297398 28015 ts_tablet_manager.cc:1398] T aedd72fd3aeb48d68f26a042069cebab P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240502 14:14:29.297706 28015 tablet_bootstrap.cc:492] T d7f02f897ce54fefb03832da3c2d95e0 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:29.297974 28013 tablet_bootstrap.cc:492] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:29.299247 28015 tablet_bootstrap.cc:654] T d7f02f897ce54fefb03832da3c2d95e0 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:29.300293 28013 tablet_bootstrap.cc:654] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:29.312691 28013 tablet_bootstrap.cc:492] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:29.312891 28013 ts_tablet_manager.cc:1367] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.006s
I20240502 14:14:29.313584 28013 raft_consensus.cc:348] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:29.313791 28013 raft_consensus.cc:374] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:29.313875 28013 raft_consensus.cc:729] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:29.314139 28013 consensus_queue.cc:260] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:29.314327 28015 tablet_bootstrap.cc:492] T d7f02f897ce54fefb03832da3c2d95e0 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:29.314467 28015 ts_tablet_manager.cc:1367] T d7f02f897ce54fefb03832da3c2d95e0 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.012s
I20240502 14:14:29.314719 28013 ts_tablet_manager.cc:1398] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:14:29.315027 28015 raft_consensus.cc:348] T d7f02f897ce54fefb03832da3c2d95e0 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:29.315167 28015 raft_consensus.cc:374] T d7f02f897ce54fefb03832da3c2d95e0 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:29.315263 28015 raft_consensus.cc:729] T d7f02f897ce54fefb03832da3c2d95e0 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:29.315479 28015 consensus_queue.cc:260] T d7f02f897ce54fefb03832da3c2d95e0 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:29.316036 28015 ts_tablet_manager.cc:1398] T d7f02f897ce54fefb03832da3c2d95e0 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:14:29.527314 27580 raft_consensus.cc:482] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:29.527561 27580 raft_consensus.cc:504] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:29.528234 27580 leader_election.cc:288] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:29.528793 26687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7f02f897ce54fefb03832da3c2d95e0" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:14:29.528815 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7f02f897ce54fefb03832da3c2d95e0" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:14:29.529012 26687 raft_consensus.cc:2454] T d7f02f897ce54fefb03832da3c2d95e0 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 0.
I20240502 14:14:29.529052 26818 raft_consensus.cc:2454] T d7f02f897ce54fefb03832da3c2d95e0 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 0.
I20240502 14:14:29.529500 26494 leader_election.cc:302] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:29.529790 27580 raft_consensus.cc:2781] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:29.529896 27580 raft_consensus.cc:482] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:29.530004 27580 raft_consensus.cc:3037] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:29.531836 27580 raft_consensus.cc:504] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:29.532438 27580 leader_election.cc:288] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Requested vote from peers 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:29.532825 26687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7f02f897ce54fefb03832da3c2d95e0" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:14:29.532871 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7f02f897ce54fefb03832da3c2d95e0" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:14:29.533049 26687 raft_consensus.cc:3037] T d7f02f897ce54fefb03832da3c2d95e0 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:29.533053 26818 raft_consensus.cc:3037] T d7f02f897ce54fefb03832da3c2d95e0 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:29.534845 26687 raft_consensus.cc:2454] T d7f02f897ce54fefb03832da3c2d95e0 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:14:29.534915 26818 raft_consensus.cc:2454] T d7f02f897ce54fefb03832da3c2d95e0 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:14:29.535259 26494 leader_election.cc:302] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:29.535567 27580 raft_consensus.cc:2781] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:29.535925 27580 raft_consensus.cc:686] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Becoming Leader. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Running, Role: LEADER
I20240502 14:14:29.536175 27580 consensus_queue.cc:237] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:29.540098 26281 catalog_manager.cc:5484] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a reported cstate change: term changed from 0 to 1, leader changed from <none> to ba187e38a2754d60a12de9ac4856430a (127.19.94.194). New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:14:29.557430 28005 raft_consensus.cc:482] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:29.557677 28005 raft_consensus.cc:504] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:29.558395 28005 leader_election.cc:288] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:14:29.558825 28005 raft_consensus.cc:482] T aedd72fd3aeb48d68f26a042069cebab P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:29.558840 26687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f3cbedfa1da42698615a544d9dc3276" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:14:29.558863 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f3cbedfa1da42698615a544d9dc3276" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:29.559048 26687 raft_consensus.cc:2454] T 9f3cbedfa1da42698615a544d9dc3276 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:14:29.558995 28005 raft_consensus.cc:504] T aedd72fd3aeb48d68f26a042069cebab P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:29.559109 26558 raft_consensus.cc:2454] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:14:29.559543 26362 leader_election.cc:302] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:29.559623 28005 leader_election.cc:288] T aedd72fd3aeb48d68f26a042069cebab P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:14:29.559854 28005 raft_consensus.cc:2781] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:29.559957 28005 raft_consensus.cc:482] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:29.560073 28005 raft_consensus.cc:3037] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:29.560285 26687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aedd72fd3aeb48d68f26a042069cebab" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:14:29.560312 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aedd72fd3aeb48d68f26a042069cebab" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:14:29.560511 26687 raft_consensus.cc:2454] T aedd72fd3aeb48d68f26a042069cebab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:14:29.560552 26818 raft_consensus.cc:2454] T aedd72fd3aeb48d68f26a042069cebab P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:14:29.560883 26363 leader_election.cc:302] T aedd72fd3aeb48d68f26a042069cebab P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:14:29.561165 27950 raft_consensus.cc:2781] T aedd72fd3aeb48d68f26a042069cebab P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:29.561280 27950 raft_consensus.cc:482] T aedd72fd3aeb48d68f26a042069cebab P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:29.561396 27950 raft_consensus.cc:3037] T aedd72fd3aeb48d68f26a042069cebab P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:29.563037 28005 raft_consensus.cc:504] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:29.563761 28005 leader_election.cc:288] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Requested vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:14:29.564168 26687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f3cbedfa1da42698615a544d9dc3276" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:14:29.564167 27950 raft_consensus.cc:504] T aedd72fd3aeb48d68f26a042069cebab P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:29.564349 26687 raft_consensus.cc:3037] T 9f3cbedfa1da42698615a544d9dc3276 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:29.564627 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f3cbedfa1da42698615a544d9dc3276" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:14:29.564844 27950 leader_election.cc:288] T aedd72fd3aeb48d68f26a042069cebab P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Requested vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:14:29.564852 26558 raft_consensus.cc:3037] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:29.565510 26688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aedd72fd3aeb48d68f26a042069cebab" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:14:29.565522 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aedd72fd3aeb48d68f26a042069cebab" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:14:29.565714 26818 raft_consensus.cc:3037] T aedd72fd3aeb48d68f26a042069cebab P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:29.565728 26688 raft_consensus.cc:3037] T aedd72fd3aeb48d68f26a042069cebab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:29.567301 26687 raft_consensus.cc:2454] T 9f3cbedfa1da42698615a544d9dc3276 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:14:29.567682 26363 leader_election.cc:302] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:14:29.567682 26558 raft_consensus.cc:2454] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:14:29.568544 26688 raft_consensus.cc:2454] T aedd72fd3aeb48d68f26a042069cebab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:14:29.568579 26818 raft_consensus.cc:2454] T aedd72fd3aeb48d68f26a042069cebab P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:14:29.568910 27950 raft_consensus.cc:2781] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:29.568953 26363 leader_election.cc:302] T aedd72fd3aeb48d68f26a042069cebab P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:29.569092 27950 raft_consensus.cc:686] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Becoming Leader. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Running, Role: LEADER
I20240502 14:14:29.569264 28005 raft_consensus.cc:2781] T aedd72fd3aeb48d68f26a042069cebab P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:29.569696 28005 raft_consensus.cc:686] T aedd72fd3aeb48d68f26a042069cebab P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Becoming Leader. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Running, Role: LEADER
I20240502 14:14:29.569804 27950 consensus_queue.cc:237] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:29.569921 28005 consensus_queue.cc:237] T aedd72fd3aeb48d68f26a042069cebab P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:29.573217 26281 catalog_manager.cc:5484] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: term changed from 0 to 1, leader changed from <none> to 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193). New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:14:29.578176 26281 catalog_manager.cc:5484] T aedd72fd3aeb48d68f26a042069cebab P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: term changed from 0 to 1, leader changed from <none> to 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193). New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
W20240502 14:14:29.957262 28030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:29.957592 28030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:29.966399 28030 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:14:29.994349 28005 consensus_queue.cc:1035] T aedd72fd3aeb48d68f26a042069cebab P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_7'
I20240502 14:14:30.000299 28005 consensus_queue.cc:1035] T aedd72fd3aeb48d68f26a042069cebab P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:30.013634 27580 consensus_queue.cc:1035] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:30.022738 27580 consensus_queue.cc:1035] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:30.122148 27950 consensus_queue.cc:1035] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:30.140967 28070 consensus_queue.cc:1035] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:30.236141 26362 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.194:36025, user_credentials={real_user=slave}} blocked reactor thread for 57908.7us
INSERT report
rows total: 2000
time total: 261.337 ms
time per row: 0.130669 ms
I20240502 14:14:30.404150 26280 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:30.414098 26537 tablet_service.cc:1467] Processing CreateTablet for tablet 11f5b70d258c445a8d3b09bd5cd46965 (DEFAULT_TABLE table=rebalance_test_table_8 [id=a0c39b03bebf4af19fe58980960cc466]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:30.414327 26406 tablet_service.cc:1467] Processing CreateTablet for tablet 11f5b70d258c445a8d3b09bd5cd46965 (DEFAULT_TABLE table=rebalance_test_table_8 [id=a0c39b03bebf4af19fe58980960cc466]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:30.414636 26537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11f5b70d258c445a8d3b09bd5cd46965. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:30.414788 26406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11f5b70d258c445a8d3b09bd5cd46965. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:30.415313 26669 tablet_service.cc:1467] Processing CreateTablet for tablet 56dddccc99524329bf2992a14c6235c7 (DEFAULT_TABLE table=rebalance_test_table_8 [id=a0c39b03bebf4af19fe58980960cc466]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:30.415743 26669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56dddccc99524329bf2992a14c6235c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:30.416532 26405 tablet_service.cc:1467] Processing CreateTablet for tablet 56dddccc99524329bf2992a14c6235c7 (DEFAULT_TABLE table=rebalance_test_table_8 [id=a0c39b03bebf4af19fe58980960cc466]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:30.417002 26405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56dddccc99524329bf2992a14c6235c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:30.417198 26407 tablet_service.cc:1467] Processing CreateTablet for tablet 64af9832dbc0464d8eea5f434172ee1f (DEFAULT_TABLE table=rebalance_test_table_8 [id=a0c39b03bebf4af19fe58980960cc466]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:30.417639 26407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64af9832dbc0464d8eea5f434172ee1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:30.420941 26798 tablet_service.cc:1467] Processing CreateTablet for tablet 11f5b70d258c445a8d3b09bd5cd46965 (DEFAULT_TABLE table=rebalance_test_table_8 [id=a0c39b03bebf4af19fe58980960cc466]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:30.421471 26798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11f5b70d258c445a8d3b09bd5cd46965. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:30.425350 26536 tablet_service.cc:1467] Processing CreateTablet for tablet 56dddccc99524329bf2992a14c6235c7 (DEFAULT_TABLE table=rebalance_test_table_8 [id=a0c39b03bebf4af19fe58980960cc466]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:30.425908 26536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56dddccc99524329bf2992a14c6235c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:30.425865 26667 tablet_service.cc:1467] Processing CreateTablet for tablet 64af9832dbc0464d8eea5f434172ee1f (DEFAULT_TABLE table=rebalance_test_table_8 [id=a0c39b03bebf4af19fe58980960cc466]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:30.426338 26667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64af9832dbc0464d8eea5f434172ee1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:30.429214 26797 tablet_service.cc:1467] Processing CreateTablet for tablet 64af9832dbc0464d8eea5f434172ee1f (DEFAULT_TABLE table=rebalance_test_table_8 [id=a0c39b03bebf4af19fe58980960cc466]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:30.429688 26797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64af9832dbc0464d8eea5f434172ee1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:30.444816 28096 tablet_bootstrap.cc:492] T 11f5b70d258c445a8d3b09bd5cd46965 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:30.445242 28098 tablet_bootstrap.cc:492] T 11f5b70d258c445a8d3b09bd5cd46965 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:30.446070 28097 tablet_bootstrap.cc:492] T 64af9832dbc0464d8eea5f434172ee1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:30.446220 28095 tablet_bootstrap.cc:492] T 11f5b70d258c445a8d3b09bd5cd46965 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:30.447434 28096 tablet_bootstrap.cc:654] T 11f5b70d258c445a8d3b09bd5cd46965 P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:30.448211 28095 tablet_bootstrap.cc:654] T 11f5b70d258c445a8d3b09bd5cd46965 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:30.448529 28097 tablet_bootstrap.cc:654] T 64af9832dbc0464d8eea5f434172ee1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:30.449098 28098 tablet_bootstrap.cc:654] T 11f5b70d258c445a8d3b09bd5cd46965 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:30.455219 28095 tablet_bootstrap.cc:492] T 11f5b70d258c445a8d3b09bd5cd46965 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:30.455428 28095 ts_tablet_manager.cc:1367] T 11f5b70d258c445a8d3b09bd5cd46965 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20240502 14:14:30.455425 28098 tablet_bootstrap.cc:492] T 11f5b70d258c445a8d3b09bd5cd46965 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:30.455483 28097 tablet_bootstrap.cc:492] T 64af9832dbc0464d8eea5f434172ee1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:30.455636 28097 ts_tablet_manager.cc:1367] T 64af9832dbc0464d8eea5f434172ee1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20240502 14:14:30.455708 28098 ts_tablet_manager.cc:1367] T 11f5b70d258c445a8d3b09bd5cd46965 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20240502 14:14:30.456152 28096 tablet_bootstrap.cc:492] T 11f5b70d258c445a8d3b09bd5cd46965 P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:30.456312 28096 ts_tablet_manager.cc:1367] T 11f5b70d258c445a8d3b09bd5cd46965 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20240502 14:14:30.456414 28095 raft_consensus.cc:348] T 11f5b70d258c445a8d3b09bd5cd46965 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:30.456527 28098 raft_consensus.cc:348] T 11f5b70d258c445a8d3b09bd5cd46965 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:30.456496 28097 raft_consensus.cc:348] T 64af9832dbc0464d8eea5f434172ee1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:30.456645 28095 raft_consensus.cc:374] T 11f5b70d258c445a8d3b09bd5cd46965 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:30.456713 28098 raft_consensus.cc:374] T 11f5b70d258c445a8d3b09bd5cd46965 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:30.456713 28097 raft_consensus.cc:374] T 64af9832dbc0464d8eea5f434172ee1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:30.456728 28095 raft_consensus.cc:729] T 11f5b70d258c445a8d3b09bd5cd46965 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:30.456815 28098 raft_consensus.cc:729] T 11f5b70d258c445a8d3b09bd5cd46965 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:30.456809 28097 raft_consensus.cc:729] T 64af9832dbc0464d8eea5f434172ee1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:30.456991 28095 consensus_queue.cc:260] T 11f5b70d258c445a8d3b09bd5cd46965 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:30.457098 28098 consensus_queue.cc:260] T 11f5b70d258c445a8d3b09bd5cd46965 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:30.457231 28096 raft_consensus.cc:348] T 11f5b70d258c445a8d3b09bd5cd46965 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:30.457479 28096 raft_consensus.cc:374] T 11f5b70d258c445a8d3b09bd5cd46965 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:30.457605 28096 raft_consensus.cc:729] T 11f5b70d258c445a8d3b09bd5cd46965 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:30.457660 28098 ts_tablet_manager.cc:1398] T 11f5b70d258c445a8d3b09bd5cd46965 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:30.457788 28095 ts_tablet_manager.cc:1398] T 11f5b70d258c445a8d3b09bd5cd46965 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:30.457998 28098 tablet_bootstrap.cc:492] T 64af9832dbc0464d8eea5f434172ee1f P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:30.457892 28096 consensus_queue.cc:260] T 11f5b70d258c445a8d3b09bd5cd46965 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:30.458199 28095 tablet_bootstrap.cc:492] T 56dddccc99524329bf2992a14c6235c7 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:30.459574 28096 ts_tablet_manager.cc:1398] T 11f5b70d258c445a8d3b09bd5cd46965 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 14:14:30.459764 28098 tablet_bootstrap.cc:654] T 64af9832dbc0464d8eea5f434172ee1f P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:30.459914 28096 tablet_bootstrap.cc:492] T 56dddccc99524329bf2992a14c6235c7 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:30.460301 28095 tablet_bootstrap.cc:654] T 56dddccc99524329bf2992a14c6235c7 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:30.461655 28096 tablet_bootstrap.cc:654] T 56dddccc99524329bf2992a14c6235c7 P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:30.469501 28095 tablet_bootstrap.cc:492] T 56dddccc99524329bf2992a14c6235c7 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:30.469874 28095 ts_tablet_manager.cc:1367] T 56dddccc99524329bf2992a14c6235c7 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20240502 14:14:30.471112 28095 raft_consensus.cc:348] T 56dddccc99524329bf2992a14c6235c7 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:30.471446 28095 raft_consensus.cc:374] T 56dddccc99524329bf2992a14c6235c7 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:30.471575 28095 raft_consensus.cc:729] T 56dddccc99524329bf2992a14c6235c7 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:30.471648 28096 tablet_bootstrap.cc:492] T 56dddccc99524329bf2992a14c6235c7 P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:30.471926 28096 ts_tablet_manager.cc:1367] T 56dddccc99524329bf2992a14c6235c7 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.008s
I20240502 14:14:30.471926 28095 consensus_queue.cc:260] T 56dddccc99524329bf2992a14c6235c7 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:30.472673 28095 ts_tablet_manager.cc:1398] T 56dddccc99524329bf2992a14c6235c7 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:14:30.472851 28096 raft_consensus.cc:348] T 56dddccc99524329bf2992a14c6235c7 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:30.473042 28095 tablet_bootstrap.cc:492] T 64af9832dbc0464d8eea5f434172ee1f P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:30.473177 28096 raft_consensus.cc:374] T 56dddccc99524329bf2992a14c6235c7 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:30.473309 28096 raft_consensus.cc:729] T 56dddccc99524329bf2992a14c6235c7 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:30.473649 28096 consensus_queue.cc:260] T 56dddccc99524329bf2992a14c6235c7 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:30.474499 28096 ts_tablet_manager.cc:1398] T 56dddccc99524329bf2992a14c6235c7 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:30.474897 28095 tablet_bootstrap.cc:654] T 64af9832dbc0464d8eea5f434172ee1f P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:30.479557 28097 consensus_queue.cc:260] T 64af9832dbc0464d8eea5f434172ee1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:30.480094 27953 raft_consensus.cc:482] T 64af9832dbc0464d8eea5f434172ee1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:30.480257 27953 raft_consensus.cc:504] T 64af9832dbc0464d8eea5f434172ee1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:30.481401 27953 leader_election.cc:288] T 64af9832dbc0464d8eea5f434172ee1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:14:30.481985 28097 ts_tablet_manager.cc:1398] T 64af9832dbc0464d8eea5f434172ee1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20240502 14:14:30.482116 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64af9832dbc0464d8eea5f434172ee1f" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:14:30.482506 28097 tablet_bootstrap.cc:492] T 56dddccc99524329bf2992a14c6235c7 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
W20240502 14:14:30.484827 26625 leader_election.cc:341] T 64af9832dbc0464d8eea5f434172ee1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:30.484973 28097 tablet_bootstrap.cc:654] T 56dddccc99524329bf2992a14c6235c7 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:30.485204 26427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64af9832dbc0464d8eea5f434172ee1f" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:14:30.485780 28026 raft_consensus.cc:482] T 11f5b70d258c445a8d3b09bd5cd46965 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:30.486001 28026 raft_consensus.cc:504] T 11f5b70d258c445a8d3b09bd5cd46965 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:30.487040 28026 leader_election.cc:288] T 11f5b70d258c445a8d3b09bd5cd46965 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:30.488040 26427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f5b70d258c445a8d3b09bd5cd46965" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
W20240502 14:14:30.489513 26624 leader_election.cc:341] T 64af9832dbc0464d8eea5f434172ee1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:30.489740 26624 leader_election.cc:302] T 64af9832dbc0464d8eea5f434172ee1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27
I20240502 14:14:30.490110 27953 raft_consensus.cc:2726] T 64af9832dbc0464d8eea5f434172ee1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:14:30.488381 26427 raft_consensus.cc:2454] T 11f5b70d258c445a8d3b09bd5cd46965 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 0.
I20240502 14:14:30.489282 28070 raft_consensus.cc:482] T 56dddccc99524329bf2992a14c6235c7 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:30.491122 28070 raft_consensus.cc:504] T 56dddccc99524329bf2992a14c6235c7 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:30.491307 28095 tablet_bootstrap.cc:492] T 64af9832dbc0464d8eea5f434172ee1f P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:30.491521 28095 ts_tablet_manager.cc:1367] T 64af9832dbc0464d8eea5f434172ee1f P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.002s
I20240502 14:14:30.492206 28070 leader_election.cc:288] T 56dddccc99524329bf2992a14c6235c7 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:14:30.492655 26493 leader_election.cc:302] T 11f5b70d258c445a8d3b09bd5cd46965 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:30.492708 28095 raft_consensus.cc:348] T 64af9832dbc0464d8eea5f434172ee1f P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:30.492959 28095 raft_consensus.cc:374] T 64af9832dbc0464d8eea5f434172ee1f P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:30.493121 28095 raft_consensus.cc:729] T 64af9832dbc0464d8eea5f434172ee1f P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:30.493407 28095 consensus_queue.cc:260] T 64af9832dbc0464d8eea5f434172ee1f P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:30.493634 26688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56dddccc99524329bf2992a14c6235c7" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:14:30.493642 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f5b70d258c445a8d3b09bd5cd46965" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:14:30.493876 26818 raft_consensus.cc:2454] T 11f5b70d258c445a8d3b09bd5cd46965 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 0.
I20240502 14:14:30.494154 28095 ts_tablet_manager.cc:1398] T 64af9832dbc0464d8eea5f434172ee1f P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 14:14:30.494602 28026 raft_consensus.cc:2781] T 11f5b70d258c445a8d3b09bd5cd46965 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:30.494748 28026 raft_consensus.cc:482] T 11f5b70d258c445a8d3b09bd5cd46965 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:30.494890 28026 raft_consensus.cc:3037] T 11f5b70d258c445a8d3b09bd5cd46965 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:30.495028 28098 tablet_bootstrap.cc:492] T 64af9832dbc0464d8eea5f434172ee1f P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:30.495157 28098 ts_tablet_manager.cc:1367] T 64af9832dbc0464d8eea5f434172ee1f P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.037s user 0.024s sys 0.012s
I20240502 14:14:30.495097 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56dddccc99524329bf2992a14c6235c7" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:30.495327 26558 raft_consensus.cc:2454] T 56dddccc99524329bf2992a14c6235c7 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:14:30.495848 28098 raft_consensus.cc:348] T 64af9832dbc0464d8eea5f434172ee1f P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:30.496085 28098 raft_consensus.cc:374] T 64af9832dbc0464d8eea5f434172ee1f P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:30.496169 28098 raft_consensus.cc:729] T 64af9832dbc0464d8eea5f434172ee1f P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
W20240502 14:14:30.496142 26363 leader_election.cc:341] T 56dddccc99524329bf2992a14c6235c7 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:30.496420 26362 leader_election.cc:302] T 56dddccc99524329bf2992a14c6235c7 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, ba187e38a2754d60a12de9ac4856430a; no voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23
I20240502 14:14:30.496443 28098 consensus_queue.cc:260] T 64af9832dbc0464d8eea5f434172ee1f P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:30.496757 28082 raft_consensus.cc:2781] T 56dddccc99524329bf2992a14c6235c7 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:30.496860 28082 raft_consensus.cc:482] T 56dddccc99524329bf2992a14c6235c7 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:30.496956 28082 raft_consensus.cc:3037] T 56dddccc99524329bf2992a14c6235c7 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:30.497074 28098 ts_tablet_manager.cc:1398] T 64af9832dbc0464d8eea5f434172ee1f P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:14:30.500382 28026 raft_consensus.cc:504] T 11f5b70d258c445a8d3b09bd5cd46965 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:30.501103 28082 raft_consensus.cc:504] T 56dddccc99524329bf2992a14c6235c7 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:30.502388 28082 leader_election.cc:288] T 56dddccc99524329bf2992a14c6235c7 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Requested vote from peers 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:14:30.502580 26688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56dddccc99524329bf2992a14c6235c7" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
W20240502 14:14:30.503320 26363 leader_election.cc:341] T 56dddccc99524329bf2992a14c6235c7 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:30.503269 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56dddccc99524329bf2992a14c6235c7" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:14:30.506444 26558 raft_consensus.cc:3037] T 56dddccc99524329bf2992a14c6235c7 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:30.508692 26558 raft_consensus.cc:2454] T 56dddccc99524329bf2992a14c6235c7 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:14:30.509243 26362 leader_election.cc:302] T 56dddccc99524329bf2992a14c6235c7 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, ba187e38a2754d60a12de9ac4856430a; no voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23
I20240502 14:14:30.509848 28082 raft_consensus.cc:2781] T 56dddccc99524329bf2992a14c6235c7 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:30.509985 26427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f5b70d258c445a8d3b09bd5cd46965" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:14:30.510172 28082 raft_consensus.cc:686] T 56dddccc99524329bf2992a14c6235c7 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Becoming Leader. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Running, Role: LEADER
I20240502 14:14:30.510241 26427 raft_consensus.cc:3037] T 11f5b70d258c445a8d3b09bd5cd46965 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:30.510699 28082 consensus_queue.cc:237] T 56dddccc99524329bf2992a14c6235c7 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:30.514318 26427 raft_consensus.cc:2454] T 11f5b70d258c445a8d3b09bd5cd46965 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:14:30.515368 26493 leader_election.cc:302] T 11f5b70d258c445a8d3b09bd5cd46965 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:30.515877 27580 raft_consensus.cc:2781] T 11f5b70d258c445a8d3b09bd5cd46965 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:30.517290 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f5b70d258c445a8d3b09bd5cd46965" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:14:30.517603 26818 raft_consensus.cc:3037] T 11f5b70d258c445a8d3b09bd5cd46965 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:30.517671 27863 raft_consensus.cc:482] T 64af9832dbc0464d8eea5f434172ee1f P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:30.517829 27863 raft_consensus.cc:504] T 64af9832dbc0464d8eea5f434172ee1f P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:30.518965 27863 leader_election.cc:288] T 64af9832dbc0464d8eea5f434172ee1f P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:14:30.520201 26688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64af9832dbc0464d8eea5f434172ee1f" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:14:30.520468 26688 raft_consensus.cc:2454] T 64af9832dbc0464d8eea5f434172ee1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 0.
I20240502 14:14:30.520967 26755 leader_election.cc:302] T 64af9832dbc0464d8eea5f434172ee1f P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:30.521262 26818 raft_consensus.cc:2454] T 11f5b70d258c445a8d3b09bd5cd46965 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:14:30.521374 27863 raft_consensus.cc:2781] T 64af9832dbc0464d8eea5f434172ee1f P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:30.521463 27863 raft_consensus.cc:482] T 64af9832dbc0464d8eea5f434172ee1f P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:30.521577 27863 raft_consensus.cc:3037] T 64af9832dbc0464d8eea5f434172ee1f P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:30.522222 26427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64af9832dbc0464d8eea5f434172ee1f" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:14:30.522554 26427 raft_consensus.cc:2454] T 64af9832dbc0464d8eea5f434172ee1f P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 0.
I20240502 14:14:30.522755 28026 leader_election.cc:288] T 11f5b70d258c445a8d3b09bd5cd46965 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Requested vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:30.524642 27863 raft_consensus.cc:504] T 64af9832dbc0464d8eea5f434172ee1f P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:30.525055 28097 tablet_bootstrap.cc:492] T 56dddccc99524329bf2992a14c6235c7 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:30.525286 28097 ts_tablet_manager.cc:1367] T 56dddccc99524329bf2992a14c6235c7 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.043s user 0.010s sys 0.028s
I20240502 14:14:30.526021 26688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64af9832dbc0464d8eea5f434172ee1f" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:14:30.526234 26688 raft_consensus.cc:3037] T 64af9832dbc0464d8eea5f434172ee1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:30.527787 28097 raft_consensus.cc:348] T 56dddccc99524329bf2992a14c6235c7 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:30.528002 28097 raft_consensus.cc:374] T 56dddccc99524329bf2992a14c6235c7 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:30.528075 28097 raft_consensus.cc:729] T 56dddccc99524329bf2992a14c6235c7 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:30.528342 28097 consensus_queue.cc:260] T 56dddccc99524329bf2992a14c6235c7 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:30.528918 28097 ts_tablet_manager.cc:1398] T 56dddccc99524329bf2992a14c6235c7 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:14:30.529259 27580 raft_consensus.cc:686] T 11f5b70d258c445a8d3b09bd5cd46965 P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Becoming Leader. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Running, Role: LEADER
I20240502 14:14:30.529894 27863 leader_election.cc:288] T 64af9832dbc0464d8eea5f434172ee1f P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Requested vote from peers 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:14:30.530040 26688 raft_consensus.cc:2454] T 64af9832dbc0464d8eea5f434172ee1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
I20240502 14:14:30.530409 26427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64af9832dbc0464d8eea5f434172ee1f" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:14:30.530550 26755 leader_election.cc:302] T 64af9832dbc0464d8eea5f434172ee1f P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:30.530640 26427 raft_consensus.cc:3037] T 64af9832dbc0464d8eea5f434172ee1f P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:30.531105 27580 consensus_queue.cc:237] T 11f5b70d258c445a8d3b09bd5cd46965 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:30.531913 26278 catalog_manager.cc:5484] T 56dddccc99524329bf2992a14c6235c7 P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: term changed from 0 to 1, leader changed from <none> to 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193). New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:14:30.532408 27863 raft_consensus.cc:2781] T 64af9832dbc0464d8eea5f434172ee1f P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:30.533784 26427 raft_consensus.cc:2454] T 64af9832dbc0464d8eea5f434172ee1f P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
I20240502 14:14:30.534642 27863 raft_consensus.cc:686] T 64af9832dbc0464d8eea5f434172ee1f P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Becoming Leader. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Running, Role: LEADER
I20240502 14:14:30.535259 27863 consensus_queue.cc:237] T 64af9832dbc0464d8eea5f434172ee1f P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:30.537420 26278 catalog_manager.cc:5484] T 11f5b70d258c445a8d3b09bd5cd46965 P ba187e38a2754d60a12de9ac4856430a reported cstate change: term changed from 0 to 1, leader changed from <none> to ba187e38a2754d60a12de9ac4856430a (127.19.94.194). New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:14:30.540338 26281 catalog_manager.cc:5484] T 64af9832dbc0464d8eea5f434172ee1f P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196). New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:14:30.571848 27953 raft_consensus.cc:482] T 56dddccc99524329bf2992a14c6235c7 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:30.572068 27953 raft_consensus.cc:504] T 56dddccc99524329bf2992a14c6235c7 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:30.572875 27953 leader_election.cc:288] T 56dddccc99524329bf2992a14c6235c7 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:14:30.573596 26427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56dddccc99524329bf2992a14c6235c7" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:14:30.573724 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56dddccc99524329bf2992a14c6235c7" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:30.574045 26558 raft_consensus.cc:2379] T 56dddccc99524329bf2992a14c6235c7 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in current term 1: Already voted for candidate 89af7f423cf44717a08d6b142f0ed49d in this term.
I20240502 14:14:30.574692 26624 leader_election.cc:302] T 56dddccc99524329bf2992a14c6235c7 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters: 89af7f423cf44717a08d6b142f0ed49d, ba187e38a2754d60a12de9ac4856430a
I20240502 14:14:30.575044 27953 raft_consensus.cc:3037] T 56dddccc99524329bf2992a14c6235c7 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:30.577245 27953 raft_consensus.cc:2726] T 56dddccc99524329bf2992a14c6235c7 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 14:14:30.933591 28113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:30.933921 28113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:30.942042 28113 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:14:30.942909 28082 consensus_queue.cc:1035] T 56dddccc99524329bf2992a14c6235c7 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:30.945371 27580 consensus_queue.cc:1035] T 11f5b70d258c445a8d3b09bd5cd46965 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:30.955127 28026 consensus_queue.cc:1035] T 11f5b70d258c445a8d3b09bd5cd46965 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:30.970844 28082 consensus_queue.cc:1035] T 56dddccc99524329bf2992a14c6235c7 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_8'
I20240502 14:14:31.093179 26688 raft_consensus.cc:1261] T 64af9832dbc0464d8eea5f434172ee1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:31.096112 27863 consensus_queue.cc:1035] T 64af9832dbc0464d8eea5f434172ee1f P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:31.105065 26427 raft_consensus.cc:1261] T 64af9832dbc0464d8eea5f434172ee1f P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:31.106833 27863 consensus_queue.cc:1035] T 64af9832dbc0464d8eea5f434172ee1f P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:31.169498 28121 mvcc.cc:205] Tried to move back new op lower bound from 7023244374389620736 to 7023244372119990272. Current Snapshot: MvccSnapshot[applied={T|T < 7023244374389620736}]
I20240502 14:14:31.220268 28124 mvcc.cc:205] Tried to move back new op lower bound from 7023244374595436544 to 7023244372119990272. Current Snapshot: MvccSnapshot[applied={T|T < 7023244374389620736}]
INSERT report
rows total: 2000
time total: 254.105 ms
time per row: 0.127052 ms
I20240502 14:14:31.375115 26281 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:31.384102 26798 tablet_service.cc:1467] Processing CreateTablet for tablet 2108a04310804945b368bcb5f88170e2 (DEFAULT_TABLE table=rebalance_test_table_9 [id=3c3eeaa25da94395a642aa569b652348]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:31.384219 26667 tablet_service.cc:1467] Processing CreateTablet for tablet 2108a04310804945b368bcb5f88170e2 (DEFAULT_TABLE table=rebalance_test_table_9 [id=3c3eeaa25da94395a642aa569b652348]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:31.384778 26667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2108a04310804945b368bcb5f88170e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:31.384912 26798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2108a04310804945b368bcb5f88170e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:31.385092 26669 tablet_service.cc:1467] Processing CreateTablet for tablet 310aa6ad3604487589dbe96afa25bb05 (DEFAULT_TABLE table=rebalance_test_table_9 [id=3c3eeaa25da94395a642aa569b652348]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:31.385948 26669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 310aa6ad3604487589dbe96afa25bb05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:31.388824 26536 tablet_service.cc:1467] Processing CreateTablet for tablet 310aa6ad3604487589dbe96afa25bb05 (DEFAULT_TABLE table=rebalance_test_table_9 [id=3c3eeaa25da94395a642aa569b652348]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:31.388870 26538 tablet_service.cc:1467] Processing CreateTablet for tablet 825ae3e90e214a6f98394765edf6ff2d (DEFAULT_TABLE table=rebalance_test_table_9 [id=3c3eeaa25da94395a642aa569b652348]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:31.389189 26406 tablet_service.cc:1467] Processing CreateTablet for tablet 825ae3e90e214a6f98394765edf6ff2d (DEFAULT_TABLE table=rebalance_test_table_9 [id=3c3eeaa25da94395a642aa569b652348]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:31.389315 26536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 310aa6ad3604487589dbe96afa25bb05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:31.389256 26537 tablet_service.cc:1467] Processing CreateTablet for tablet 2108a04310804945b368bcb5f88170e2 (DEFAULT_TABLE table=rebalance_test_table_9 [id=3c3eeaa25da94395a642aa569b652348]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:31.389565 26538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 825ae3e90e214a6f98394765edf6ff2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:31.389605 26406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 825ae3e90e214a6f98394765edf6ff2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:31.389775 26537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2108a04310804945b368bcb5f88170e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:31.393371 26797 tablet_service.cc:1467] Processing CreateTablet for tablet 310aa6ad3604487589dbe96afa25bb05 (DEFAULT_TABLE table=rebalance_test_table_9 [id=3c3eeaa25da94395a642aa569b652348]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:31.393385 26799 tablet_service.cc:1467] Processing CreateTablet for tablet 825ae3e90e214a6f98394765edf6ff2d (DEFAULT_TABLE table=rebalance_test_table_9 [id=3c3eeaa25da94395a642aa569b652348]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:31.393932 26797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 310aa6ad3604487589dbe96afa25bb05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:31.397528 26799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 825ae3e90e214a6f98394765edf6ff2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:31.405000 28181 tablet_bootstrap.cc:492] T 310aa6ad3604487589dbe96afa25bb05 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:31.405647 28182 tablet_bootstrap.cc:492] T 2108a04310804945b368bcb5f88170e2 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:31.407554 28182 tablet_bootstrap.cc:654] T 2108a04310804945b368bcb5f88170e2 P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:31.407668 28181 tablet_bootstrap.cc:654] T 310aa6ad3604487589dbe96afa25bb05 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:31.412958 28186 tablet_bootstrap.cc:492] T 825ae3e90e214a6f98394765edf6ff2d P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:31.414019 28182 tablet_bootstrap.cc:492] T 2108a04310804945b368bcb5f88170e2 P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:31.414259 28182 ts_tablet_manager.cc:1367] T 2108a04310804945b368bcb5f88170e2 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240502 14:14:31.414999 28186 tablet_bootstrap.cc:654] T 825ae3e90e214a6f98394765edf6ff2d P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:31.415139 28182 raft_consensus.cc:348] T 2108a04310804945b368bcb5f88170e2 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:31.415886 28182 raft_consensus.cc:374] T 2108a04310804945b368bcb5f88170e2 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:31.416041 28182 raft_consensus.cc:729] T 2108a04310804945b368bcb5f88170e2 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:31.416498 28182 consensus_queue.cc:260] T 2108a04310804945b368bcb5f88170e2 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:31.416976 28183 tablet_bootstrap.cc:492] T 2108a04310804945b368bcb5f88170e2 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:31.417311 28181 tablet_bootstrap.cc:492] T 310aa6ad3604487589dbe96afa25bb05 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:31.417699 28181 ts_tablet_manager.cc:1367] T 310aa6ad3604487589dbe96afa25bb05 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.006s
I20240502 14:14:31.418604 28181 raft_consensus.cc:348] T 310aa6ad3604487589dbe96afa25bb05 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:31.418920 28181 raft_consensus.cc:374] T 310aa6ad3604487589dbe96afa25bb05 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:31.419076 28181 raft_consensus.cc:729] T 310aa6ad3604487589dbe96afa25bb05 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:31.419430 28181 consensus_queue.cc:260] T 310aa6ad3604487589dbe96afa25bb05 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:31.420208 28183 tablet_bootstrap.cc:654] T 2108a04310804945b368bcb5f88170e2 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:31.421125 28181 ts_tablet_manager.cc:1398] T 310aa6ad3604487589dbe96afa25bb05 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:14:31.421527 28181 tablet_bootstrap.cc:492] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:31.423929 28182 ts_tablet_manager.cc:1398] T 2108a04310804945b368bcb5f88170e2 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 14:14:31.424376 28186 tablet_bootstrap.cc:492] T 825ae3e90e214a6f98394765edf6ff2d P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:31.424571 28186 ts_tablet_manager.cc:1367] T 825ae3e90e214a6f98394765edf6ff2d P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20240502 14:14:31.425166 28181 tablet_bootstrap.cc:654] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:31.425597 28186 raft_consensus.cc:348] T 825ae3e90e214a6f98394765edf6ff2d P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:31.425937 28186 raft_consensus.cc:374] T 825ae3e90e214a6f98394765edf6ff2d P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:31.426090 28186 raft_consensus.cc:729] T 825ae3e90e214a6f98394765edf6ff2d P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:31.426458 28186 consensus_queue.cc:260] T 825ae3e90e214a6f98394765edf6ff2d P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:31.427153 28186 ts_tablet_manager.cc:1398] T 825ae3e90e214a6f98394765edf6ff2d P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:31.427872 28182 tablet_bootstrap.cc:492] T 825ae3e90e214a6f98394765edf6ff2d P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:31.429729 28182 tablet_bootstrap.cc:654] T 825ae3e90e214a6f98394765edf6ff2d P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:31.431252 28183 tablet_bootstrap.cc:492] T 2108a04310804945b368bcb5f88170e2 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:31.431370 28181 tablet_bootstrap.cc:492] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:31.431427 28183 ts_tablet_manager.cc:1367] T 2108a04310804945b368bcb5f88170e2 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.007s
I20240502 14:14:31.431617 28181 ts_tablet_manager.cc:1367] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s
I20240502 14:14:31.432639 28181 raft_consensus.cc:348] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:31.433008 28181 raft_consensus.cc:374] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:31.433136 28181 raft_consensus.cc:729] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:31.433467 28181 consensus_queue.cc:260] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:31.434090 28183 raft_consensus.cc:348] T 2108a04310804945b368bcb5f88170e2 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:31.434217 28181 ts_tablet_manager.cc:1398] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:31.434371 28183 raft_consensus.cc:374] T 2108a04310804945b368bcb5f88170e2 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:31.434506 28183 raft_consensus.cc:729] T 2108a04310804945b368bcb5f88170e2 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:31.434738 28183 consensus_queue.cc:260] T 2108a04310804945b368bcb5f88170e2 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:31.436276 28183 ts_tablet_manager.cc:1398] T 2108a04310804945b368bcb5f88170e2 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:14:31.436631 28183 tablet_bootstrap.cc:492] T 825ae3e90e214a6f98394765edf6ff2d P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:31.438814 28183 tablet_bootstrap.cc:654] T 825ae3e90e214a6f98394765edf6ff2d P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:31.440191 28182 tablet_bootstrap.cc:492] T 825ae3e90e214a6f98394765edf6ff2d P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:31.440415 28182 ts_tablet_manager.cc:1367] T 825ae3e90e214a6f98394765edf6ff2d P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20240502 14:14:31.440848 28183 tablet_bootstrap.cc:492] T 825ae3e90e214a6f98394765edf6ff2d P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:31.441022 28183 ts_tablet_manager.cc:1367] T 825ae3e90e214a6f98394765edf6ff2d P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:14:31.441278 28182 raft_consensus.cc:348] T 825ae3e90e214a6f98394765edf6ff2d P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:31.441684 28182 raft_consensus.cc:374] T 825ae3e90e214a6f98394765edf6ff2d P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:31.441826 28182 raft_consensus.cc:729] T 825ae3e90e214a6f98394765edf6ff2d P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:31.441864 28183 raft_consensus.cc:348] T 825ae3e90e214a6f98394765edf6ff2d P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:31.442255 28183 raft_consensus.cc:374] T 825ae3e90e214a6f98394765edf6ff2d P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:31.442157 28182 consensus_queue.cc:260] T 825ae3e90e214a6f98394765edf6ff2d P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:31.442373 28183 raft_consensus.cc:729] T 825ae3e90e214a6f98394765edf6ff2d P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:31.442881 28182 ts_tablet_manager.cc:1398] T 825ae3e90e214a6f98394765edf6ff2d P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:14:31.442845 28183 consensus_queue.cc:260] T 825ae3e90e214a6f98394765edf6ff2d P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:31.443310 28182 tablet_bootstrap.cc:492] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:31.443576 28183 ts_tablet_manager.cc:1398] T 825ae3e90e214a6f98394765edf6ff2d P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:31.443976 28183 tablet_bootstrap.cc:492] T 310aa6ad3604487589dbe96afa25bb05 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:31.445375 28183 tablet_bootstrap.cc:654] T 310aa6ad3604487589dbe96afa25bb05 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:31.446535 28182 tablet_bootstrap.cc:654] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:31.447351 28183 tablet_bootstrap.cc:492] T 310aa6ad3604487589dbe96afa25bb05 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:31.447566 28183 ts_tablet_manager.cc:1367] T 310aa6ad3604487589dbe96afa25bb05 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:14:31.448429 28183 raft_consensus.cc:348] T 310aa6ad3604487589dbe96afa25bb05 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:31.448661 28183 raft_consensus.cc:374] T 310aa6ad3604487589dbe96afa25bb05 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:31.448773 28183 raft_consensus.cc:729] T 310aa6ad3604487589dbe96afa25bb05 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:31.449061 28183 consensus_queue.cc:260] T 310aa6ad3604487589dbe96afa25bb05 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:31.449657 28183 ts_tablet_manager.cc:1398] T 310aa6ad3604487589dbe96afa25bb05 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:14:31.450112 28182 tablet_bootstrap.cc:492] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:31.450264 28182 ts_tablet_manager.cc:1367] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20240502 14:14:31.450873 28182 raft_consensus.cc:348] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:31.451032 28182 raft_consensus.cc:374] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:31.451131 28182 raft_consensus.cc:729] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:31.451332 28182 consensus_queue.cc:260] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:31.451864 28182 ts_tablet_manager.cc:1398] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:14:31.553985 28109 raft_consensus.cc:482] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:31.554167 28109 raft_consensus.cc:504] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:31.554785 28109 leader_election.cc:288] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:31.555224 26688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "310aa6ad3604487589dbe96afa25bb05" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:14:31.555420 26688 raft_consensus.cc:2454] T 310aa6ad3604487589dbe96afa25bb05 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 0.
I20240502 14:14:31.555437 26817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "310aa6ad3604487589dbe96afa25bb05" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:14:31.555691 26817 raft_consensus.cc:2454] T 310aa6ad3604487589dbe96afa25bb05 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 0.
I20240502 14:14:31.556102 26494 leader_election.cc:302] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:31.556432 28109 raft_consensus.cc:2781] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:31.556560 28109 raft_consensus.cc:482] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:31.556666 28109 raft_consensus.cc:3037] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:31.558775 28109 raft_consensus.cc:504] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:31.559288 28109 leader_election.cc:288] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Requested vote from peers 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:31.559717 26688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "310aa6ad3604487589dbe96afa25bb05" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:14:31.559773 26817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "310aa6ad3604487589dbe96afa25bb05" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:14:31.559911 26688 raft_consensus.cc:3037] T 310aa6ad3604487589dbe96afa25bb05 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:31.559937 26817 raft_consensus.cc:3037] T 310aa6ad3604487589dbe96afa25bb05 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:31.561836 26688 raft_consensus.cc:2454] T 310aa6ad3604487589dbe96afa25bb05 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:14:31.561892 26817 raft_consensus.cc:2454] T 310aa6ad3604487589dbe96afa25bb05 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:14:31.562250 26494 leader_election.cc:302] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:31.562563 28109 raft_consensus.cc:2781] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:31.562822 28109 raft_consensus.cc:686] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Becoming Leader. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Running, Role: LEADER
I20240502 14:14:31.563086 28109 consensus_queue.cc:237] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:31.566413 26284 catalog_manager.cc:5484] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a reported cstate change: term changed from 0 to 1, leader changed from <none> to ba187e38a2754d60a12de9ac4856430a (127.19.94.194). New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:14:31.600968 27953 raft_consensus.cc:482] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:31.601310 27953 raft_consensus.cc:504] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:31.602082 27953 leader_election.cc:288] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:14:31.602634 26817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2108a04310804945b368bcb5f88170e2" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:14:31.602741 26557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2108a04310804945b368bcb5f88170e2" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:31.602876 26817 raft_consensus.cc:2454] T 2108a04310804945b368bcb5f88170e2 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:14:31.603039 26557 raft_consensus.cc:2454] T 2108a04310804945b368bcb5f88170e2 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:14:31.603403 26625 leader_election.cc:302] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:31.603785 27953 raft_consensus.cc:2781] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:31.603891 27953 raft_consensus.cc:482] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:31.604005 27953 raft_consensus.cc:3037] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:31.606002 27953 raft_consensus.cc:504] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:31.606644 27953 leader_election.cc:288] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Requested vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:14:31.607081 26817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2108a04310804945b368bcb5f88170e2" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:14:31.607156 26557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2108a04310804945b368bcb5f88170e2" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:14:31.607292 26817 raft_consensus.cc:3037] T 2108a04310804945b368bcb5f88170e2 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:31.607420 26557 raft_consensus.cc:3037] T 2108a04310804945b368bcb5f88170e2 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:31.609297 26817 raft_consensus.cc:2454] T 2108a04310804945b368bcb5f88170e2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:14:31.609519 26557 raft_consensus.cc:2454] T 2108a04310804945b368bcb5f88170e2 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:14:31.609736 26625 leader_election.cc:302] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:31.610014 27953 raft_consensus.cc:2781] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:31.610414 27953 raft_consensus.cc:686] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Becoming Leader. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Running, Role: LEADER
I20240502 14:14:31.610740 27953 consensus_queue.cc:237] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:31.614145 26284 catalog_manager.cc:5484] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195). New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:14:31.631613 28109 raft_consensus.cc:482] T 825ae3e90e214a6f98394765edf6ff2d P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:31.631834 28109 raft_consensus.cc:504] T 825ae3e90e214a6f98394765edf6ff2d P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:31.632184 28176 raft_consensus.cc:482] T 825ae3e90e214a6f98394765edf6ff2d P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:31.632359 28176 raft_consensus.cc:504] T 825ae3e90e214a6f98394765edf6ff2d P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:31.632580 28109 leader_election.cc:288] T 825ae3e90e214a6f98394765edf6ff2d P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:31.633139 28176 leader_election.cc:288] T 825ae3e90e214a6f98394765edf6ff2d P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:31.633196 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "825ae3e90e214a6f98394765edf6ff2d" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:14:31.633212 26817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "825ae3e90e214a6f98394765edf6ff2d" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:14:31.633415 26426 raft_consensus.cc:2454] T 825ae3e90e214a6f98394765edf6ff2d P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 0.
I20240502 14:14:31.633455 26817 raft_consensus.cc:2454] T 825ae3e90e214a6f98394765edf6ff2d P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 0.
I20240502 14:14:31.633848 26557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "825ae3e90e214a6f98394765edf6ff2d" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:31.634140 26557 raft_consensus.cc:2454] T 825ae3e90e214a6f98394765edf6ff2d P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:14:31.634233 26817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "825ae3e90e214a6f98394765edf6ff2d" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:14:31.634447 26817 raft_consensus.cc:2454] T 825ae3e90e214a6f98394765edf6ff2d P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:14:31.633957 26494 leader_election.cc:302] T 825ae3e90e214a6f98394765edf6ff2d P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:31.634951 28109 raft_consensus.cc:2781] T 825ae3e90e214a6f98394765edf6ff2d P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:31.634912 26363 leader_election.cc:302] T 825ae3e90e214a6f98394765edf6ff2d P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:31.635043 28109 raft_consensus.cc:482] T 825ae3e90e214a6f98394765edf6ff2d P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:31.635138 28109 raft_consensus.cc:3037] T 825ae3e90e214a6f98394765edf6ff2d P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:31.635243 28176 raft_consensus.cc:2781] T 825ae3e90e214a6f98394765edf6ff2d P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:31.635349 28176 raft_consensus.cc:482] T 825ae3e90e214a6f98394765edf6ff2d P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:31.635447 28176 raft_consensus.cc:3037] T 825ae3e90e214a6f98394765edf6ff2d P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:31.638147 28109 raft_consensus.cc:504] T 825ae3e90e214a6f98394765edf6ff2d P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:31.638233 28176 raft_consensus.cc:504] T 825ae3e90e214a6f98394765edf6ff2d P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:31.639011 28109 leader_election.cc:288] T 825ae3e90e214a6f98394765edf6ff2d P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Requested vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:31.639544 26557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "825ae3e90e214a6f98394765edf6ff2d" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:14:31.639544 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "825ae3e90e214a6f98394765edf6ff2d" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:14:31.639797 26557 raft_consensus.cc:2379] T 825ae3e90e214a6f98394765edf6ff2d P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 89af7f423cf44717a08d6b142f0ed49d in current term 1: Already voted for candidate ba187e38a2754d60a12de9ac4856430a in this term.
I20240502 14:14:31.639838 26426 raft_consensus.cc:2379] T 825ae3e90e214a6f98394765edf6ff2d P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ba187e38a2754d60a12de9ac4856430a in current term 1: Already voted for candidate 89af7f423cf44717a08d6b142f0ed49d in this term.
I20240502 14:14:31.640175 26817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "825ae3e90e214a6f98394765edf6ff2d" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:14:31.640347 28176 leader_election.cc:288] T 825ae3e90e214a6f98394765edf6ff2d P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Requested vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:31.640409 26817 raft_consensus.cc:3037] T 825ae3e90e214a6f98394765edf6ff2d P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:31.641016 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "825ae3e90e214a6f98394765edf6ff2d" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:14:31.641551 26363 leader_election.cc:302] T 825ae3e90e214a6f98394765edf6ff2d P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d; no voters: ba187e38a2754d60a12de9ac4856430a, e4cfb8fc5a474177af6848a6efa13a27
I20240502 14:14:31.641880 28176 raft_consensus.cc:2726] T 825ae3e90e214a6f98394765edf6ff2d P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 14:14:31.642925 26817 raft_consensus.cc:2454] T 825ae3e90e214a6f98394765edf6ff2d P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:14:31.643316 26494 leader_election.cc:302] T 825ae3e90e214a6f98394765edf6ff2d P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a, e4cfb8fc5a474177af6848a6efa13a27; no voters: 89af7f423cf44717a08d6b142f0ed49d
I20240502 14:14:31.643637 28109 raft_consensus.cc:2781] T 825ae3e90e214a6f98394765edf6ff2d P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:31.643815 28109 raft_consensus.cc:686] T 825ae3e90e214a6f98394765edf6ff2d P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Becoming Leader. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Running, Role: LEADER
I20240502 14:14:31.644105 28109 consensus_queue.cc:237] T 825ae3e90e214a6f98394765edf6ff2d P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:31.647166 26284 catalog_manager.cc:5484] T 825ae3e90e214a6f98394765edf6ff2d P ba187e38a2754d60a12de9ac4856430a reported cstate change: term changed from 0 to 1, leader changed from <none> to ba187e38a2754d60a12de9ac4856430a (127.19.94.194). New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:14:32.071242 28109 consensus_queue.cc:1035] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:32.076481 28109 consensus_queue.cc:1035] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:32.079892 28199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:32.080361 28199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:32.090241 28199 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:14:32.114246 28026 consensus_queue.cc:1035] T 825ae3e90e214a6f98394765edf6ff2d P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:32.121393 28216 consensus_queue.cc:1035] T 825ae3e90e214a6f98394765edf6ff2d P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_9'
I20240502 14:14:32.193372 27953 consensus_queue.cc:1035] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:32.226652 28231 consensus_queue.cc:1035] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 329.764 ms
time per row: 0.164882 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 14:14:33.539834 26282 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:33.549680 26667 tablet_service.cc:1467] Processing CreateTablet for tablet c3ca60f2b4fa4852a9ec585c640fd21e (DEFAULT_TABLE table=rebalance_test_table_10 [id=37d9dbdf887c42b984b9623e7fec549d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:33.550194 26667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3ca60f2b4fa4852a9ec585c640fd21e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:33.550161 26537 tablet_service.cc:1467] Processing CreateTablet for tablet c3ca60f2b4fa4852a9ec585c640fd21e (DEFAULT_TABLE table=rebalance_test_table_10 [id=37d9dbdf887c42b984b9623e7fec549d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:33.550599 26537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3ca60f2b4fa4852a9ec585c640fd21e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:33.551440 26406 tablet_service.cc:1467] Processing CreateTablet for tablet c3ca60f2b4fa4852a9ec585c640fd21e (DEFAULT_TABLE table=rebalance_test_table_10 [id=37d9dbdf887c42b984b9623e7fec549d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:33.551851 26406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3ca60f2b4fa4852a9ec585c640fd21e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:33.551944 26407 tablet_service.cc:1467] Processing CreateTablet for tablet 6c5232e2b8ec4aba8bf2fe03aa132a6e (DEFAULT_TABLE table=rebalance_test_table_10 [id=37d9dbdf887c42b984b9623e7fec549d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:33.552318 26407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c5232e2b8ec4aba8bf2fe03aa132a6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:33.552889 26669 tablet_service.cc:1467] Processing CreateTablet for tablet 6c5232e2b8ec4aba8bf2fe03aa132a6e (DEFAULT_TABLE table=rebalance_test_table_10 [id=37d9dbdf887c42b984b9623e7fec549d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:33.553236 26669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c5232e2b8ec4aba8bf2fe03aa132a6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:33.556758 26405 tablet_service.cc:1467] Processing CreateTablet for tablet 8d7c842c6c2e4929b07173ab4b7dc224 (DEFAULT_TABLE table=rebalance_test_table_10 [id=37d9dbdf887c42b984b9623e7fec549d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:33.556787 26536 tablet_service.cc:1467] Processing CreateTablet for tablet 8d7c842c6c2e4929b07173ab4b7dc224 (DEFAULT_TABLE table=rebalance_test_table_10 [id=37d9dbdf887c42b984b9623e7fec549d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:33.556777 26668 tablet_service.cc:1467] Processing CreateTablet for tablet 8d7c842c6c2e4929b07173ab4b7dc224 (DEFAULT_TABLE table=rebalance_test_table_10 [id=37d9dbdf887c42b984b9623e7fec549d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:33.557133 26536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d7c842c6c2e4929b07173ab4b7dc224. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:33.557142 26405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d7c842c6c2e4929b07173ab4b7dc224. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:33.557165 26668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d7c842c6c2e4929b07173ab4b7dc224. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:33.561344 26798 tablet_service.cc:1467] Processing CreateTablet for tablet 6c5232e2b8ec4aba8bf2fe03aa132a6e (DEFAULT_TABLE table=rebalance_test_table_10 [id=37d9dbdf887c42b984b9623e7fec549d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:33.561779 26798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c5232e2b8ec4aba8bf2fe03aa132a6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:33.571297 28270 tablet_bootstrap.cc:492] T 8d7c842c6c2e4929b07173ab4b7dc224 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:33.571691 28269 tablet_bootstrap.cc:492] T 8d7c842c6c2e4929b07173ab4b7dc224 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:33.574015 28270 tablet_bootstrap.cc:654] T 8d7c842c6c2e4929b07173ab4b7dc224 P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:33.574733 28271 tablet_bootstrap.cc:492] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:33.575471 28268 tablet_bootstrap.cc:492] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:33.575686 28269 tablet_bootstrap.cc:654] T 8d7c842c6c2e4929b07173ab4b7dc224 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:33.576483 28271 tablet_bootstrap.cc:654] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:33.577338 28268 tablet_bootstrap.cc:654] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:33.581420 28268 tablet_bootstrap.cc:492] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:33.581686 28268 ts_tablet_manager.cc:1367] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240502 14:14:33.582949 28268 raft_consensus.cc:348] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:33.583400 28268 raft_consensus.cc:374] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:33.583570 28268 raft_consensus.cc:729] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:33.583993 28268 consensus_queue.cc:260] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:33.584698 28268 ts_tablet_manager.cc:1398] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:33.584839 28269 tablet_bootstrap.cc:492] T 8d7c842c6c2e4929b07173ab4b7dc224 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:33.585007 28269 ts_tablet_manager.cc:1367] T 8d7c842c6c2e4929b07173ab4b7dc224 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20240502 14:14:33.585523 28268 tablet_bootstrap.cc:492] T 8d7c842c6c2e4929b07173ab4b7dc224 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:33.586091 28270 tablet_bootstrap.cc:492] T 8d7c842c6c2e4929b07173ab4b7dc224 P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:33.585963 28269 raft_consensus.cc:348] T 8d7c842c6c2e4929b07173ab4b7dc224 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:33.586177 28269 raft_consensus.cc:374] T 8d7c842c6c2e4929b07173ab4b7dc224 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:33.586344 28270 ts_tablet_manager.cc:1367] T 8d7c842c6c2e4929b07173ab4b7dc224 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.009s
I20240502 14:14:33.586330 28269 raft_consensus.cc:729] T 8d7c842c6c2e4929b07173ab4b7dc224 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:33.586648 28269 consensus_queue.cc:260] T 8d7c842c6c2e4929b07173ab4b7dc224 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:33.587196 28270 raft_consensus.cc:348] T 8d7c842c6c2e4929b07173ab4b7dc224 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:33.587445 28269 ts_tablet_manager.cc:1398] T 8d7c842c6c2e4929b07173ab4b7dc224 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:33.587708 28271 tablet_bootstrap.cc:492] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:33.587757 28269 tablet_bootstrap.cc:492] T c3ca60f2b4fa4852a9ec585c640fd21e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:33.587949 28271 ts_tablet_manager.cc:1367] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.005s
I20240502 14:14:33.588304 28270 raft_consensus.cc:374] T 8d7c842c6c2e4929b07173ab4b7dc224 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:33.588436 28270 raft_consensus.cc:729] T 8d7c842c6c2e4929b07173ab4b7dc224 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:33.588730 28270 consensus_queue.cc:260] T 8d7c842c6c2e4929b07173ab4b7dc224 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:33.589391 28269 tablet_bootstrap.cc:654] T c3ca60f2b4fa4852a9ec585c640fd21e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:33.590982 28270 ts_tablet_manager.cc:1398] T 8d7c842c6c2e4929b07173ab4b7dc224 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 14:14:33.591351 28270 tablet_bootstrap.cc:492] T c3ca60f2b4fa4852a9ec585c640fd21e P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:33.591871 28271 raft_consensus.cc:348] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:33.592357 28271 raft_consensus.cc:374] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:33.592543 28271 raft_consensus.cc:729] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:33.592870 28271 consensus_queue.cc:260] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:33.593631 28268 tablet_bootstrap.cc:654] T 8d7c842c6c2e4929b07173ab4b7dc224 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:33.594712 28270 tablet_bootstrap.cc:654] T c3ca60f2b4fa4852a9ec585c640fd21e P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:33.596736 28271 ts_tablet_manager.cc:1398] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240502 14:14:33.598191 28169 raft_consensus.cc:482] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:33.598337 28169 raft_consensus.cc:504] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:33.599098 28169 leader_election.cc:288] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:14:33.599848 26688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c5232e2b8ec4aba8bf2fe03aa132a6e" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:14:33.601074 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c5232e2b8ec4aba8bf2fe03aa132a6e" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
W20240502 14:14:33.601153 26755 leader_election.cc:341] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:33.601392 26426 raft_consensus.cc:2454] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 0.
I20240502 14:14:33.602279 26754 leader_election.cc:302] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27; no voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23
I20240502 14:14:33.602550 28169 raft_consensus.cc:2781] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:33.602665 28169 raft_consensus.cc:482] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:33.602761 28169 raft_consensus.cc:3037] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:33.603883 28269 tablet_bootstrap.cc:492] T c3ca60f2b4fa4852a9ec585c640fd21e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:33.603883 28268 tablet_bootstrap.cc:492] T 8d7c842c6c2e4929b07173ab4b7dc224 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:33.604048 28270 tablet_bootstrap.cc:492] T c3ca60f2b4fa4852a9ec585c640fd21e P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:33.604061 28268 ts_tablet_manager.cc:1367] T 8d7c842c6c2e4929b07173ab4b7dc224 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.005s
I20240502 14:14:33.604171 28269 ts_tablet_manager.cc:1367] T c3ca60f2b4fa4852a9ec585c640fd21e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.010s
I20240502 14:14:33.604226 28270 ts_tablet_manager.cc:1367] T c3ca60f2b4fa4852a9ec585c640fd21e P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.008s
I20240502 14:14:33.604887 28268 raft_consensus.cc:348] T 8d7c842c6c2e4929b07173ab4b7dc224 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:33.605010 28270 raft_consensus.cc:348] T c3ca60f2b4fa4852a9ec585c640fd21e P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:33.605250 28270 raft_consensus.cc:374] T c3ca60f2b4fa4852a9ec585c640fd21e P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:33.605192 28269 raft_consensus.cc:348] T c3ca60f2b4fa4852a9ec585c640fd21e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:33.605345 28270 raft_consensus.cc:729] T c3ca60f2b4fa4852a9ec585c640fd21e P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:33.605350 28268 raft_consensus.cc:374] T 8d7c842c6c2e4929b07173ab4b7dc224 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:33.605441 28269 raft_consensus.cc:374] T c3ca60f2b4fa4852a9ec585c640fd21e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:33.605574 28269 raft_consensus.cc:729] T c3ca60f2b4fa4852a9ec585c640fd21e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:33.605536 28169 raft_consensus.cc:504] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:33.605626 28270 consensus_queue.cc:260] T c3ca60f2b4fa4852a9ec585c640fd21e P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:33.605868 28268 raft_consensus.cc:729] T 8d7c842c6c2e4929b07173ab4b7dc224 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:33.605844 28269 consensus_queue.cc:260] T c3ca60f2b4fa4852a9ec585c640fd21e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:33.607120 28270 ts_tablet_manager.cc:1398] T c3ca60f2b4fa4852a9ec585c640fd21e P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:14:33.608561 28169 leader_election.cc:288] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Requested vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:14:33.608704 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c5232e2b8ec4aba8bf2fe03aa132a6e" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:14:33.608937 26426 raft_consensus.cc:3037] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:33.609251 26688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c5232e2b8ec4aba8bf2fe03aa132a6e" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
W20240502 14:14:33.609761 26755 leader_election.cc:341] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:33.609746 28268 consensus_queue.cc:260] T 8d7c842c6c2e4929b07173ab4b7dc224 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:33.610306 28268 ts_tablet_manager.cc:1398] T 8d7c842c6c2e4929b07173ab4b7dc224 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:14:33.610706 28268 tablet_bootstrap.cc:492] T c3ca60f2b4fa4852a9ec585c640fd21e P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:33.611382 28269 ts_tablet_manager.cc:1398] T c3ca60f2b4fa4852a9ec585c640fd21e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.007s user 0.002s sys 0.002s
I20240502 14:14:33.611661 28269 tablet_bootstrap.cc:492] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:33.612125 26426 raft_consensus.cc:2454] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
I20240502 14:14:33.612448 28268 tablet_bootstrap.cc:654] T c3ca60f2b4fa4852a9ec585c640fd21e P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:33.613091 26754 leader_election.cc:302] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27; no voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23
I20240502 14:14:33.613432 28269 tablet_bootstrap.cc:654] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:33.615888 28169 raft_consensus.cc:2781] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:33.624754 28169 raft_consensus.cc:686] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Becoming Leader. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Running, Role: LEADER
I20240502 14:14:33.625362 28169 consensus_queue.cc:237] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:33.630647 28269 tablet_bootstrap.cc:492] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:33.630672 28268 tablet_bootstrap.cc:492] T c3ca60f2b4fa4852a9ec585c640fd21e P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:33.630839 28268 ts_tablet_manager.cc:1367] T c3ca60f2b4fa4852a9ec585c640fd21e P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.012s
I20240502 14:14:33.630847 28269 ts_tablet_manager.cc:1367] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.010s
I20240502 14:14:33.631618 28268 raft_consensus.cc:348] T c3ca60f2b4fa4852a9ec585c640fd21e P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:33.631587 28269 raft_consensus.cc:348] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:33.631793 28268 raft_consensus.cc:374] T c3ca60f2b4fa4852a9ec585c640fd21e P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:33.631803 28269 raft_consensus.cc:374] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:33.631893 28269 raft_consensus.cc:729] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:33.631911 28268 raft_consensus.cc:729] T c3ca60f2b4fa4852a9ec585c640fd21e P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:33.632134 28269 consensus_queue.cc:260] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:33.632411 28268 consensus_queue.cc:260] T c3ca60f2b4fa4852a9ec585c640fd21e P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:33.632763 28269 ts_tablet_manager.cc:1398] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:33.632525 26282 catalog_manager.cc:5484] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196). New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:14:33.633011 28268 ts_tablet_manager.cc:1398] T c3ca60f2b4fa4852a9ec585c640fd21e P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:14:33.676455 28279 raft_consensus.cc:482] T c3ca60f2b4fa4852a9ec585c640fd21e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:33.676645 28279 raft_consensus.cc:504] T c3ca60f2b4fa4852a9ec585c640fd21e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:33.677245 28279 leader_election.cc:288] T c3ca60f2b4fa4852a9ec585c640fd21e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:14:33.677825 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3ca60f2b4fa4852a9ec585c640fd21e" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:14:33.677942 26557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3ca60f2b4fa4852a9ec585c640fd21e" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:33.678083 26426 raft_consensus.cc:2454] T c3ca60f2b4fa4852a9ec585c640fd21e P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:14:33.678231 26557 raft_consensus.cc:2454] T c3ca60f2b4fa4852a9ec585c640fd21e P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:14:33.678544 26624 leader_election.cc:302] T c3ca60f2b4fa4852a9ec585c640fd21e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:14:33.678848 28279 raft_consensus.cc:2781] T c3ca60f2b4fa4852a9ec585c640fd21e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:33.678972 28279 raft_consensus.cc:482] T c3ca60f2b4fa4852a9ec585c640fd21e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:33.679069 28279 raft_consensus.cc:3037] T c3ca60f2b4fa4852a9ec585c640fd21e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:33.681138 28279 raft_consensus.cc:504] T c3ca60f2b4fa4852a9ec585c640fd21e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:33.681727 28279 leader_election.cc:288] T c3ca60f2b4fa4852a9ec585c640fd21e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Requested vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:14:33.682200 26557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3ca60f2b4fa4852a9ec585c640fd21e" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:14:33.682206 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3ca60f2b4fa4852a9ec585c640fd21e" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:14:33.682382 26557 raft_consensus.cc:3037] T c3ca60f2b4fa4852a9ec585c640fd21e P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:33.682436 26426 raft_consensus.cc:3037] T c3ca60f2b4fa4852a9ec585c640fd21e P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:33.684332 26557 raft_consensus.cc:2454] T c3ca60f2b4fa4852a9ec585c640fd21e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:14:33.684384 26426 raft_consensus.cc:2454] T c3ca60f2b4fa4852a9ec585c640fd21e P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:14:33.684741 26624 leader_election.cc:302] T c3ca60f2b4fa4852a9ec585c640fd21e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:33.685034 28279 raft_consensus.cc:2781] T c3ca60f2b4fa4852a9ec585c640fd21e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:33.685209 28279 raft_consensus.cc:686] T c3ca60f2b4fa4852a9ec585c640fd21e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Becoming Leader. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Running, Role: LEADER
I20240502 14:14:33.685562 28279 consensus_queue.cc:237] T c3ca60f2b4fa4852a9ec585c640fd21e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:33.688637 26282 catalog_manager.cc:5484] T c3ca60f2b4fa4852a9ec585c640fd21e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195). New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:14:33.728065 28279 raft_consensus.cc:482] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:33.728313 28279 raft_consensus.cc:504] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:33.728950 28279 leader_election.cc:288] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:33.729570 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c5232e2b8ec4aba8bf2fe03aa132a6e" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:14:33.729568 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c5232e2b8ec4aba8bf2fe03aa132a6e" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:14:33.729848 26426 raft_consensus.cc:2379] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in current term 1: Already voted for candidate e4cfb8fc5a474177af6848a6efa13a27 in this term.
I20240502 14:14:33.730268 26625 leader_election.cc:302] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27
I20240502 14:14:33.730597 28279 raft_consensus.cc:3037] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:33.732587 28279 raft_consensus.cc:2726] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:14:33.795825 28263 raft_consensus.cc:482] T 8d7c842c6c2e4929b07173ab4b7dc224 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:33.796051 28263 raft_consensus.cc:504] T 8d7c842c6c2e4929b07173ab4b7dc224 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:33.796769 28263 leader_election.cc:288] T 8d7c842c6c2e4929b07173ab4b7dc224 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:14:33.797339 26688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d7c842c6c2e4929b07173ab4b7dc224" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:14:33.797372 26557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d7c842c6c2e4929b07173ab4b7dc224" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:33.797569 26688 raft_consensus.cc:2454] T 8d7c842c6c2e4929b07173ab4b7dc224 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:14:33.797600 26557 raft_consensus.cc:2454] T 8d7c842c6c2e4929b07173ab4b7dc224 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:14:33.798008 26363 leader_election.cc:302] T 8d7c842c6c2e4929b07173ab4b7dc224 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:14:33.798302 28263 raft_consensus.cc:2781] T 8d7c842c6c2e4929b07173ab4b7dc224 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:33.798419 28263 raft_consensus.cc:482] T 8d7c842c6c2e4929b07173ab4b7dc224 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:33.798493 28263 raft_consensus.cc:3037] T 8d7c842c6c2e4929b07173ab4b7dc224 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:33.800382 28263 raft_consensus.cc:504] T 8d7c842c6c2e4929b07173ab4b7dc224 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:33.801034 28263 leader_election.cc:288] T 8d7c842c6c2e4929b07173ab4b7dc224 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Requested vote from peers 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:14:33.801498 26688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d7c842c6c2e4929b07173ab4b7dc224" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:14:33.801533 26557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d7c842c6c2e4929b07173ab4b7dc224" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:14:33.801779 26688 raft_consensus.cc:3037] T 8d7c842c6c2e4929b07173ab4b7dc224 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:33.801784 26557 raft_consensus.cc:3037] T 8d7c842c6c2e4929b07173ab4b7dc224 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:33.804883 26688 raft_consensus.cc:2454] T 8d7c842c6c2e4929b07173ab4b7dc224 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:14:33.804878 26557 raft_consensus.cc:2454] T 8d7c842c6c2e4929b07173ab4b7dc224 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:14:33.805423 26362 leader_election.cc:302] T 8d7c842c6c2e4929b07173ab4b7dc224 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:33.805895 28263 raft_consensus.cc:2781] T 8d7c842c6c2e4929b07173ab4b7dc224 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:33.806044 28263 raft_consensus.cc:686] T 8d7c842c6c2e4929b07173ab4b7dc224 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Becoming Leader. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Running, Role: LEADER
I20240502 14:14:33.806377 28263 consensus_queue.cc:237] T 8d7c842c6c2e4929b07173ab4b7dc224 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:33.810771 26282 catalog_manager.cc:5484] T 8d7c842c6c2e4929b07173ab4b7dc224 P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: term changed from 0 to 1, leader changed from <none> to 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193). New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:14:34.087936 28279 consensus_queue.cc:1035] T c3ca60f2b4fa4852a9ec585c640fd21e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:34.091277 28169 consensus_queue.cc:1035] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:34.100348 28279 consensus_queue.cc:1035] T c3ca60f2b4fa4852a9ec585c640fd21e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:34.114842 28282 consensus_queue.cc:1035] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:34.241343 28288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:34.241693 28288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:34.253013 28288 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
I20240502 14:14:34.330333 28263 consensus_queue.cc:1035] T 8d7c842c6c2e4929b07173ab4b7dc224 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:34.335789 28176 consensus_queue.cc:1035] T 8d7c842c6c2e4929b07173ab4b7dc224 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:34.551731 26363 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 79078.2us
INSERT report
rows total: 2000
time total: 277.736 ms
time per row: 0.138868 ms
I20240502 14:14:34.702064 26282 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:34.730288 26669 tablet_service.cc:1467] Processing CreateTablet for tablet 4df2bd4e182c4fe991da385d92d3f76f (DEFAULT_TABLE table=rebalance_test_table_11 [id=c7861c8f192f462e8ed2198ec9c0c5ea]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:34.730703 26797 tablet_service.cc:1467] Processing CreateTablet for tablet 4df2bd4e182c4fe991da385d92d3f76f (DEFAULT_TABLE table=rebalance_test_table_11 [id=c7861c8f192f462e8ed2198ec9c0c5ea]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:34.730837 26669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4df2bd4e182c4fe991da385d92d3f76f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:34.730815 26799 tablet_service.cc:1467] Processing CreateTablet for tablet 0290ebbcf609427cae8fce6637019336 (DEFAULT_TABLE table=rebalance_test_table_11 [id=c7861c8f192f462e8ed2198ec9c0c5ea]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:34.731098 26797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4df2bd4e182c4fe991da385d92d3f76f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:34.731626 26799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0290ebbcf609427cae8fce6637019336. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:34.733547 26798 tablet_service.cc:1467] Processing CreateTablet for tablet d1aa4f4be5894ec5a64275f1172e1f55 (DEFAULT_TABLE table=rebalance_test_table_11 [id=c7861c8f192f462e8ed2198ec9c0c5ea]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:34.733937 26798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1aa4f4be5894ec5a64275f1172e1f55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:34.736194 26537 tablet_service.cc:1467] Processing CreateTablet for tablet 4df2bd4e182c4fe991da385d92d3f76f (DEFAULT_TABLE table=rebalance_test_table_11 [id=c7861c8f192f462e8ed2198ec9c0c5ea]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:34.736618 26537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4df2bd4e182c4fe991da385d92d3f76f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:34.736852 26667 tablet_service.cc:1467] Processing CreateTablet for tablet 0290ebbcf609427cae8fce6637019336 (DEFAULT_TABLE table=rebalance_test_table_11 [id=c7861c8f192f462e8ed2198ec9c0c5ea]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:34.737232 26667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0290ebbcf609427cae8fce6637019336. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:34.738406 26407 tablet_service.cc:1467] Processing CreateTablet for tablet d1aa4f4be5894ec5a64275f1172e1f55 (DEFAULT_TABLE table=rebalance_test_table_11 [id=c7861c8f192f462e8ed2198ec9c0c5ea]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:34.738827 26407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1aa4f4be5894ec5a64275f1172e1f55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:34.740504 26536 tablet_service.cc:1467] Processing CreateTablet for tablet d1aa4f4be5894ec5a64275f1172e1f55 (DEFAULT_TABLE table=rebalance_test_table_11 [id=c7861c8f192f462e8ed2198ec9c0c5ea]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:34.740864 26536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1aa4f4be5894ec5a64275f1172e1f55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:34.746145 26406 tablet_service.cc:1467] Processing CreateTablet for tablet 0290ebbcf609427cae8fce6637019336 (DEFAULT_TABLE table=rebalance_test_table_11 [id=c7861c8f192f462e8ed2198ec9c0c5ea]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:34.746555 26406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0290ebbcf609427cae8fce6637019336. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:34.753196 28351 tablet_bootstrap.cc:492] T 0290ebbcf609427cae8fce6637019336 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:34.754102 28352 tablet_bootstrap.cc:492] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:34.756444 28351 tablet_bootstrap.cc:654] T 0290ebbcf609427cae8fce6637019336 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:34.758380 28352 tablet_bootstrap.cc:654] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:34.762835 28353 tablet_bootstrap.cc:492] T d1aa4f4be5894ec5a64275f1172e1f55 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:34.767119 28353 tablet_bootstrap.cc:654] T d1aa4f4be5894ec5a64275f1172e1f55 P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:34.770434 28352 tablet_bootstrap.cc:492] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:34.770716 28352 ts_tablet_manager.cc:1367] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20240502 14:14:34.772186 28352 raft_consensus.cc:348] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:34.772601 28352 raft_consensus.cc:374] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:34.772769 28352 raft_consensus.cc:729] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:34.773123 28352 consensus_queue.cc:260] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:34.773943 28357 tablet_bootstrap.cc:492] T d1aa4f4be5894ec5a64275f1172e1f55 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:34.774706 28352 ts_tablet_manager.cc:1398] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:14:34.776785 28357 tablet_bootstrap.cc:654] T d1aa4f4be5894ec5a64275f1172e1f55 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:34.777177 28352 tablet_bootstrap.cc:492] T 4df2bd4e182c4fe991da385d92d3f76f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:34.780496 28352 tablet_bootstrap.cc:654] T 4df2bd4e182c4fe991da385d92d3f76f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:34.780843 28353 tablet_bootstrap.cc:492] T d1aa4f4be5894ec5a64275f1172e1f55 P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:34.781015 28353 ts_tablet_manager.cc:1367] T d1aa4f4be5894ec5a64275f1172e1f55 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20240502 14:14:34.781128 28351 tablet_bootstrap.cc:492] T 0290ebbcf609427cae8fce6637019336 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:34.781749 28351 ts_tablet_manager.cc:1367] T 0290ebbcf609427cae8fce6637019336 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.000s
I20240502 14:14:34.782112 28353 raft_consensus.cc:348] T d1aa4f4be5894ec5a64275f1172e1f55 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:34.782346 28353 raft_consensus.cc:374] T d1aa4f4be5894ec5a64275f1172e1f55 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:34.782452 28353 raft_consensus.cc:729] T d1aa4f4be5894ec5a64275f1172e1f55 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:34.782737 28353 consensus_queue.cc:260] T d1aa4f4be5894ec5a64275f1172e1f55 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:34.782601 28351 raft_consensus.cc:348] T 0290ebbcf609427cae8fce6637019336 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:34.783581 28351 raft_consensus.cc:374] T 0290ebbcf609427cae8fce6637019336 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:34.783737 28351 raft_consensus.cc:729] T 0290ebbcf609427cae8fce6637019336 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:34.784296 28351 consensus_queue.cc:260] T 0290ebbcf609427cae8fce6637019336 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:34.785288 28353 ts_tablet_manager.cc:1398] T d1aa4f4be5894ec5a64275f1172e1f55 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:14:34.785630 28353 tablet_bootstrap.cc:492] T 4df2bd4e182c4fe991da385d92d3f76f P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:34.787277 28357 tablet_bootstrap.cc:492] T d1aa4f4be5894ec5a64275f1172e1f55 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:34.787501 28357 ts_tablet_manager.cc:1367] T d1aa4f4be5894ec5a64275f1172e1f55 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240502 14:14:34.787679 28353 tablet_bootstrap.cc:654] T 4df2bd4e182c4fe991da385d92d3f76f P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:34.788502 28357 raft_consensus.cc:348] T d1aa4f4be5894ec5a64275f1172e1f55 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:34.788837 28357 raft_consensus.cc:374] T d1aa4f4be5894ec5a64275f1172e1f55 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:34.788975 28357 raft_consensus.cc:729] T d1aa4f4be5894ec5a64275f1172e1f55 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:34.789305 28357 consensus_queue.cc:260] T d1aa4f4be5894ec5a64275f1172e1f55 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:34.790081 28357 ts_tablet_manager.cc:1398] T d1aa4f4be5894ec5a64275f1172e1f55 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:34.790462 28357 tablet_bootstrap.cc:492] T 0290ebbcf609427cae8fce6637019336 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:34.792469 28357 tablet_bootstrap.cc:654] T 0290ebbcf609427cae8fce6637019336 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:34.792939 28351 ts_tablet_manager.cc:1398] T 0290ebbcf609427cae8fce6637019336 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 14:14:34.793690 28351 tablet_bootstrap.cc:492] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:34.794350 28352 tablet_bootstrap.cc:492] T 4df2bd4e182c4fe991da385d92d3f76f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:34.794593 28352 ts_tablet_manager.cc:1367] T 4df2bd4e182c4fe991da385d92d3f76f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.004s
I20240502 14:14:34.795555 28352 raft_consensus.cc:348] T 4df2bd4e182c4fe991da385d92d3f76f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:34.795853 28352 raft_consensus.cc:374] T 4df2bd4e182c4fe991da385d92d3f76f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:34.795961 28352 raft_consensus.cc:729] T 4df2bd4e182c4fe991da385d92d3f76f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:34.797577 28357 tablet_bootstrap.cc:492] T 0290ebbcf609427cae8fce6637019336 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:34.797744 28357 ts_tablet_manager.cc:1367] T 0290ebbcf609427cae8fce6637019336 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240502 14:14:34.798056 28352 consensus_queue.cc:260] T 4df2bd4e182c4fe991da385d92d3f76f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:34.798573 28357 raft_consensus.cc:348] T 0290ebbcf609427cae8fce6637019336 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:34.798795 28357 raft_consensus.cc:374] T 0290ebbcf609427cae8fce6637019336 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:34.798882 28357 raft_consensus.cc:729] T 0290ebbcf609427cae8fce6637019336 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:34.798943 28351 tablet_bootstrap.cc:654] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:34.799113 28352 ts_tablet_manager.cc:1398] T 4df2bd4e182c4fe991da385d92d3f76f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20240502 14:14:34.799149 28357 consensus_queue.cc:260] T 0290ebbcf609427cae8fce6637019336 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:34.800009 28357 ts_tablet_manager.cc:1398] T 0290ebbcf609427cae8fce6637019336 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:34.806713 28353 tablet_bootstrap.cc:492] T 4df2bd4e182c4fe991da385d92d3f76f P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:34.806993 28353 ts_tablet_manager.cc:1367] T 4df2bd4e182c4fe991da385d92d3f76f P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.000s
I20240502 14:14:34.807183 28351 tablet_bootstrap.cc:492] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:34.807379 28351 ts_tablet_manager.cc:1367] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.000s
I20240502 14:14:34.808065 28353 raft_consensus.cc:348] T 4df2bd4e182c4fe991da385d92d3f76f P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:34.808210 28351 raft_consensus.cc:348] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:34.808374 28353 raft_consensus.cc:374] T 4df2bd4e182c4fe991da385d92d3f76f P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:34.808487 28353 raft_consensus.cc:729] T 4df2bd4e182c4fe991da385d92d3f76f P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:34.808511 28351 raft_consensus.cc:374] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:34.808585 28351 raft_consensus.cc:729] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:34.808806 28353 consensus_queue.cc:260] T 4df2bd4e182c4fe991da385d92d3f76f P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:34.808868 28351 consensus_queue.cc:260] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:34.809612 28353 ts_tablet_manager.cc:1398] T 4df2bd4e182c4fe991da385d92d3f76f P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:14:34.810508 28351 ts_tablet_manager.cc:1398] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 14:14:34.810830 28351 tablet_bootstrap.cc:492] T d1aa4f4be5894ec5a64275f1172e1f55 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:34.813611 28351 tablet_bootstrap.cc:654] T d1aa4f4be5894ec5a64275f1172e1f55 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:34.817095 28351 tablet_bootstrap.cc:492] T d1aa4f4be5894ec5a64275f1172e1f55 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:34.817278 28351 ts_tablet_manager.cc:1367] T d1aa4f4be5894ec5a64275f1172e1f55 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240502 14:14:34.818104 28351 raft_consensus.cc:348] T d1aa4f4be5894ec5a64275f1172e1f55 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:34.818354 28351 raft_consensus.cc:374] T d1aa4f4be5894ec5a64275f1172e1f55 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:34.818454 28351 raft_consensus.cc:729] T d1aa4f4be5894ec5a64275f1172e1f55 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:34.818696 28351 consensus_queue.cc:260] T d1aa4f4be5894ec5a64275f1172e1f55 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:34.819340 28351 ts_tablet_manager.cc:1398] T d1aa4f4be5894ec5a64275f1172e1f55 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:14:34.956272 28282 raft_consensus.cc:482] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:34.956526 28282 raft_consensus.cc:504] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:34.957230 28282 leader_election.cc:288] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:14:34.957767 26557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4df2bd4e182c4fe991da385d92d3f76f" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:34.957805 26689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4df2bd4e182c4fe991da385d92d3f76f" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:14:34.957998 26557 raft_consensus.cc:2454] T 4df2bd4e182c4fe991da385d92d3f76f P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 0.
I20240502 14:14:34.958029 26689 raft_consensus.cc:2454] T 4df2bd4e182c4fe991da385d92d3f76f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 0.
I20240502 14:14:34.958469 26754 leader_election.cc:302] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:34.958799 28282 raft_consensus.cc:2781] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:34.958889 28282 raft_consensus.cc:482] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:34.959014 28282 raft_consensus.cc:3037] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:34.961026 28282 raft_consensus.cc:504] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:34.961581 28282 leader_election.cc:288] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Requested vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:14:34.962029 26689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4df2bd4e182c4fe991da385d92d3f76f" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:14:34.962049 26557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4df2bd4e182c4fe991da385d92d3f76f" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:14:34.962240 26689 raft_consensus.cc:3037] T 4df2bd4e182c4fe991da385d92d3f76f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:34.962235 26557 raft_consensus.cc:3037] T 4df2bd4e182c4fe991da385d92d3f76f P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:34.964165 26689 raft_consensus.cc:2454] T 4df2bd4e182c4fe991da385d92d3f76f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
I20240502 14:14:34.964448 26557 raft_consensus.cc:2454] T 4df2bd4e182c4fe991da385d92d3f76f P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
I20240502 14:14:34.964610 26755 leader_election.cc:302] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:34.964941 28282 raft_consensus.cc:2781] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:34.965123 28282 raft_consensus.cc:686] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Becoming Leader. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Running, Role: LEADER
I20240502 14:14:34.965370 28282 consensus_queue.cc:237] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:34.968366 26283 catalog_manager.cc:5484] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196). New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:14:35.072680 28026 raft_consensus.cc:482] T d1aa4f4be5894ec5a64275f1172e1f55 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:35.072948 28026 raft_consensus.cc:504] T d1aa4f4be5894ec5a64275f1172e1f55 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:35.073572 28026 leader_election.cc:288] T d1aa4f4be5894ec5a64275f1172e1f55 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:14:35.074215 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1aa4f4be5894ec5a64275f1172e1f55" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:14:35.074194 26817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1aa4f4be5894ec5a64275f1172e1f55" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:14:35.074457 26426 raft_consensus.cc:2454] T d1aa4f4be5894ec5a64275f1172e1f55 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 0.
I20240502 14:14:35.074479 26817 raft_consensus.cc:2454] T d1aa4f4be5894ec5a64275f1172e1f55 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 0.
I20240502 14:14:35.074952 26493 leader_election.cc:302] T d1aa4f4be5894ec5a64275f1172e1f55 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:35.075268 28026 raft_consensus.cc:2781] T d1aa4f4be5894ec5a64275f1172e1f55 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:35.075405 28026 raft_consensus.cc:482] T d1aa4f4be5894ec5a64275f1172e1f55 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:35.075506 28026 raft_consensus.cc:3037] T d1aa4f4be5894ec5a64275f1172e1f55 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:35.077534 28026 raft_consensus.cc:504] T d1aa4f4be5894ec5a64275f1172e1f55 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:35.078135 28026 leader_election.cc:288] T d1aa4f4be5894ec5a64275f1172e1f55 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Requested vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:14:35.078689 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1aa4f4be5894ec5a64275f1172e1f55" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:14:35.078748 26817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1aa4f4be5894ec5a64275f1172e1f55" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:14:35.078897 26426 raft_consensus.cc:3037] T d1aa4f4be5894ec5a64275f1172e1f55 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:35.078948 26817 raft_consensus.cc:3037] T d1aa4f4be5894ec5a64275f1172e1f55 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:35.080770 26426 raft_consensus.cc:2454] T d1aa4f4be5894ec5a64275f1172e1f55 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:14:35.080768 26817 raft_consensus.cc:2454] T d1aa4f4be5894ec5a64275f1172e1f55 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:14:35.081172 26493 leader_election.cc:302] T d1aa4f4be5894ec5a64275f1172e1f55 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:35.081444 28026 raft_consensus.cc:2781] T d1aa4f4be5894ec5a64275f1172e1f55 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:35.081597 28026 raft_consensus.cc:686] T d1aa4f4be5894ec5a64275f1172e1f55 P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Becoming Leader. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Running, Role: LEADER
I20240502 14:14:35.081887 28026 consensus_queue.cc:237] T d1aa4f4be5894ec5a64275f1172e1f55 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:35.084791 26283 catalog_manager.cc:5484] T d1aa4f4be5894ec5a64275f1172e1f55 P ba187e38a2754d60a12de9ac4856430a reported cstate change: term changed from 0 to 1, leader changed from <none> to ba187e38a2754d60a12de9ac4856430a (127.19.94.194). New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:14:35.166796 27953 raft_consensus.cc:482] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:35.167037 27953 raft_consensus.cc:504] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:35.167752 27953 leader_election.cc:288] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:14:35.168339 26817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0290ebbcf609427cae8fce6637019336" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:14:35.168365 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0290ebbcf609427cae8fce6637019336" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:14:35.168617 26817 raft_consensus.cc:2454] T 0290ebbcf609427cae8fce6637019336 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:14:35.168637 26426 raft_consensus.cc:2454] T 0290ebbcf609427cae8fce6637019336 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:14:35.169040 26625 leader_election.cc:302] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:35.169337 27953 raft_consensus.cc:2781] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:35.169441 27953 raft_consensus.cc:482] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:35.169555 27953 raft_consensus.cc:3037] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:35.171414 27953 raft_consensus.cc:504] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:35.172040 27953 leader_election.cc:288] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Requested vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:14:35.172502 26817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0290ebbcf609427cae8fce6637019336" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:14:35.172700 26817 raft_consensus.cc:3037] T 0290ebbcf609427cae8fce6637019336 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:35.172747 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0290ebbcf609427cae8fce6637019336" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:14:35.172958 26426 raft_consensus.cc:3037] T 0290ebbcf609427cae8fce6637019336 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:35.174749 26426 raft_consensus.cc:2454] T 0290ebbcf609427cae8fce6637019336 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:14:35.174764 26817 raft_consensus.cc:2454] T 0290ebbcf609427cae8fce6637019336 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:14:35.175166 26624 leader_election.cc:302] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:14:35.175446 27953 raft_consensus.cc:2781] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:35.175607 27953 raft_consensus.cc:686] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Becoming Leader. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Running, Role: LEADER
I20240502 14:14:35.175841 27953 consensus_queue.cc:237] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:35.179638 26283 catalog_manager.cc:5484] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195). New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:14:35.381202 28282 consensus_queue.cc:1035] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:35.389189 28282 consensus_queue.cc:1035] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:35.514176 28322 consensus_queue.cc:1035] T d1aa4f4be5894ec5a64275f1172e1f55 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:35.520007 28322 consensus_queue.cc:1035] T d1aa4f4be5894ec5a64275f1172e1f55 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:35.572926 28367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:35.573403 28367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:35.581396 28367 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
I20240502 14:14:35.643304 27953 consensus_queue.cc:1035] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:35.656195 28279 consensus_queue.cc:1035] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 251.565 ms
time per row: 0.125783 ms
I20240502 14:14:35.984877 26282 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalance_test_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:35.994731 26536 tablet_service.cc:1467] Processing CreateTablet for tablet 1a147a6c2ba04d848a122021fd7c63c6 (DEFAULT_TABLE table=rebalance_test_table_12 [id=52458461122d4161bd91b41ead7538e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:35.995025 26537 tablet_service.cc:1467] Processing CreateTablet for tablet e79bc37363cc48e7943e0275981be31a (DEFAULT_TABLE table=rebalance_test_table_12 [id=52458461122d4161bd91b41ead7538e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:35.995358 26536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a147a6c2ba04d848a122021fd7c63c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:35.995641 26537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e79bc37363cc48e7943e0275981be31a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:35.995735 26406 tablet_service.cc:1467] Processing CreateTablet for tablet e79bc37363cc48e7943e0275981be31a (DEFAULT_TABLE table=rebalance_test_table_12 [id=52458461122d4161bd91b41ead7538e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:35.996178 26406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e79bc37363cc48e7943e0275981be31a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:35.996708 26538 tablet_service.cc:1467] Processing CreateTablet for tablet 2924b2da6f8347aca55042bfbf741c38 (DEFAULT_TABLE table=rebalance_test_table_12 [id=52458461122d4161bd91b41ead7538e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:35.997102 26538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2924b2da6f8347aca55042bfbf741c38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:36.001345 26407 tablet_service.cc:1467] Processing CreateTablet for tablet 1a147a6c2ba04d848a122021fd7c63c6 (DEFAULT_TABLE table=rebalance_test_table_12 [id=52458461122d4161bd91b41ead7538e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:36.001359 26799 tablet_service.cc:1467] Processing CreateTablet for tablet 2924b2da6f8347aca55042bfbf741c38 (DEFAULT_TABLE table=rebalance_test_table_12 [id=52458461122d4161bd91b41ead7538e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:35.996150 26667 tablet_service.cc:1467] Processing CreateTablet for tablet e79bc37363cc48e7943e0275981be31a (DEFAULT_TABLE table=rebalance_test_table_12 [id=52458461122d4161bd91b41ead7538e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:36.001744 26407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a147a6c2ba04d848a122021fd7c63c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:36.001811 26799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2924b2da6f8347aca55042bfbf741c38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:36.001991 26667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e79bc37363cc48e7943e0275981be31a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:36.004938 26798 tablet_service.cc:1467] Processing CreateTablet for tablet 1a147a6c2ba04d848a122021fd7c63c6 (DEFAULT_TABLE table=rebalance_test_table_12 [id=52458461122d4161bd91b41ead7538e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:36.005601 26798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a147a6c2ba04d848a122021fd7c63c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:36.009850 28429 tablet_bootstrap.cc:492] T e79bc37363cc48e7943e0275981be31a P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:36.011777 28429 tablet_bootstrap.cc:654] T e79bc37363cc48e7943e0275981be31a P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:36.013664 28431 tablet_bootstrap.cc:492] T 1a147a6c2ba04d848a122021fd7c63c6 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:36.014103 26669 tablet_service.cc:1467] Processing CreateTablet for tablet 2924b2da6f8347aca55042bfbf741c38 (DEFAULT_TABLE table=rebalance_test_table_12 [id=52458461122d4161bd91b41ead7538e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:36.014652 26669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2924b2da6f8347aca55042bfbf741c38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:36.015635 28431 tablet_bootstrap.cc:654] T 1a147a6c2ba04d848a122021fd7c63c6 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:36.019133 28429 tablet_bootstrap.cc:492] T e79bc37363cc48e7943e0275981be31a P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:36.020648 28429 ts_tablet_manager.cc:1367] T e79bc37363cc48e7943e0275981be31a P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240502 14:14:36.021755 28429 raft_consensus.cc:348] T e79bc37363cc48e7943e0275981be31a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:36.021997 28429 raft_consensus.cc:374] T e79bc37363cc48e7943e0275981be31a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:36.022137 28429 raft_consensus.cc:729] T e79bc37363cc48e7943e0275981be31a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:36.023803 28429 consensus_queue.cc:260] T e79bc37363cc48e7943e0275981be31a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:36.025627 28429 ts_tablet_manager.cc:1398] T e79bc37363cc48e7943e0275981be31a P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:14:36.031827 28429 tablet_bootstrap.cc:492] T 2924b2da6f8347aca55042bfbf741c38 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:36.034250 28429 tablet_bootstrap.cc:654] T 2924b2da6f8347aca55042bfbf741c38 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:36.039191 28430 tablet_bootstrap.cc:492] T e79bc37363cc48e7943e0275981be31a P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:36.044428 28430 tablet_bootstrap.cc:654] T e79bc37363cc48e7943e0275981be31a P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:36.049813 28430 tablet_bootstrap.cc:492] T e79bc37363cc48e7943e0275981be31a P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:36.049998 28430 ts_tablet_manager.cc:1367] T e79bc37363cc48e7943e0275981be31a P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240502 14:14:36.050660 28430 raft_consensus.cc:348] T e79bc37363cc48e7943e0275981be31a P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:36.050838 28430 raft_consensus.cc:374] T e79bc37363cc48e7943e0275981be31a P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:36.050951 28430 raft_consensus.cc:729] T e79bc37363cc48e7943e0275981be31a P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:36.051220 28430 consensus_queue.cc:260] T e79bc37363cc48e7943e0275981be31a P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:36.051769 28430 ts_tablet_manager.cc:1398] T e79bc37363cc48e7943e0275981be31a P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:36.052419 28430 tablet_bootstrap.cc:492] T 1a147a6c2ba04d848a122021fd7c63c6 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:36.052976 28435 tablet_bootstrap.cc:492] T 2924b2da6f8347aca55042bfbf741c38 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:36.055431 28435 tablet_bootstrap.cc:654] T 2924b2da6f8347aca55042bfbf741c38 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:36.059074 28430 tablet_bootstrap.cc:654] T 1a147a6c2ba04d848a122021fd7c63c6 P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:36.063681 28429 tablet_bootstrap.cc:492] T 2924b2da6f8347aca55042bfbf741c38 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:36.064177 28429 ts_tablet_manager.cc:1367] T 2924b2da6f8347aca55042bfbf741c38 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.006s
I20240502 14:14:36.065245 28431 tablet_bootstrap.cc:492] T 1a147a6c2ba04d848a122021fd7c63c6 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:36.065304 28430 tablet_bootstrap.cc:492] T 1a147a6c2ba04d848a122021fd7c63c6 P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:36.065501 28431 ts_tablet_manager.cc:1367] T 1a147a6c2ba04d848a122021fd7c63c6 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.052s user 0.018s sys 0.015s
I20240502 14:14:36.065565 28430 ts_tablet_manager.cc:1367] T 1a147a6c2ba04d848a122021fd7c63c6 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.006s
I20240502 14:14:36.066488 28429 raft_consensus.cc:348] T 2924b2da6f8347aca55042bfbf741c38 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:36.066790 28429 raft_consensus.cc:374] T 2924b2da6f8347aca55042bfbf741c38 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:36.066589 28430 raft_consensus.cc:348] T 1a147a6c2ba04d848a122021fd7c63c6 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:36.066918 28429 raft_consensus.cc:729] T 2924b2da6f8347aca55042bfbf741c38 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:36.067059 28430 raft_consensus.cc:374] T 1a147a6c2ba04d848a122021fd7c63c6 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:36.067227 28430 raft_consensus.cc:729] T 1a147a6c2ba04d848a122021fd7c63c6 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:36.067270 28429 consensus_queue.cc:260] T 2924b2da6f8347aca55042bfbf741c38 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:36.067277 28431 raft_consensus.cc:348] T 1a147a6c2ba04d848a122021fd7c63c6 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:36.067734 28431 raft_consensus.cc:374] T 1a147a6c2ba04d848a122021fd7c63c6 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:36.067894 28431 raft_consensus.cc:729] T 1a147a6c2ba04d848a122021fd7c63c6 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:36.067999 28429 ts_tablet_manager.cc:1398] T 2924b2da6f8347aca55042bfbf741c38 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 14:14:36.068224 28431 consensus_queue.cc:260] T 1a147a6c2ba04d848a122021fd7c63c6 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:36.068672 28430 consensus_queue.cc:260] T 1a147a6c2ba04d848a122021fd7c63c6 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:36.068979 28431 ts_tablet_manager.cc:1398] T 1a147a6c2ba04d848a122021fd7c63c6 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:14:36.069345 28431 tablet_bootstrap.cc:492] T e79bc37363cc48e7943e0275981be31a P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:36.070262 28430 ts_tablet_manager.cc:1398] T 1a147a6c2ba04d848a122021fd7c63c6 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 14:14:36.070657 28430 tablet_bootstrap.cc:492] T 2924b2da6f8347aca55042bfbf741c38 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:36.072357 28431 tablet_bootstrap.cc:654] T e79bc37363cc48e7943e0275981be31a P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:36.073570 28430 tablet_bootstrap.cc:654] T 2924b2da6f8347aca55042bfbf741c38 P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:36.076164 28435 tablet_bootstrap.cc:492] T 2924b2da6f8347aca55042bfbf741c38 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:36.076464 28435 ts_tablet_manager.cc:1367] T 2924b2da6f8347aca55042bfbf741c38 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.000s
I20240502 14:14:36.080466 28431 tablet_bootstrap.cc:492] T e79bc37363cc48e7943e0275981be31a P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:36.080667 28431 ts_tablet_manager.cc:1367] T e79bc37363cc48e7943e0275981be31a P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20240502 14:14:36.080768 28435 raft_consensus.cc:348] T 2924b2da6f8347aca55042bfbf741c38 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:36.081029 28435 raft_consensus.cc:374] T 2924b2da6f8347aca55042bfbf741c38 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:36.081128 28435 raft_consensus.cc:729] T 2924b2da6f8347aca55042bfbf741c38 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:36.081152 28430 tablet_bootstrap.cc:492] T 2924b2da6f8347aca55042bfbf741c38 P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:36.081298 28430 ts_tablet_manager.cc:1367] T 2924b2da6f8347aca55042bfbf741c38 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.005s
I20240502 14:14:36.081421 28435 consensus_queue.cc:260] T 2924b2da6f8347aca55042bfbf741c38 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:36.082023 28435 ts_tablet_manager.cc:1398] T 2924b2da6f8347aca55042bfbf741c38 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:14:36.082337 28435 tablet_bootstrap.cc:492] T 1a147a6c2ba04d848a122021fd7c63c6 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:36.082231 28430 raft_consensus.cc:348] T 2924b2da6f8347aca55042bfbf741c38 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:36.082441 28430 raft_consensus.cc:374] T 2924b2da6f8347aca55042bfbf741c38 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:36.082530 28430 raft_consensus.cc:729] T 2924b2da6f8347aca55042bfbf741c38 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:36.082808 28430 consensus_queue.cc:260] T 2924b2da6f8347aca55042bfbf741c38 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:36.083410 28430 ts_tablet_manager.cc:1398] T 2924b2da6f8347aca55042bfbf741c38 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:36.084210 28431 raft_consensus.cc:348] T e79bc37363cc48e7943e0275981be31a P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:36.084470 28431 raft_consensus.cc:374] T e79bc37363cc48e7943e0275981be31a P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:36.084554 28431 raft_consensus.cc:729] T e79bc37363cc48e7943e0275981be31a P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:36.084785 28431 consensus_queue.cc:260] T e79bc37363cc48e7943e0275981be31a P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:36.085424 28431 ts_tablet_manager.cc:1398] T e79bc37363cc48e7943e0275981be31a P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240502 14:14:36.095073 28279 raft_consensus.cc:482] T 2924b2da6f8347aca55042bfbf741c38 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:36.095173 28435 tablet_bootstrap.cc:654] T 1a147a6c2ba04d848a122021fd7c63c6 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:36.095293 28279 raft_consensus.cc:504] T 2924b2da6f8347aca55042bfbf741c38 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:36.096066 28279 leader_election.cc:288] T 2924b2da6f8347aca55042bfbf741c38 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:36.096592 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2924b2da6f8347aca55042bfbf741c38" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:36.096675 26817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2924b2da6f8347aca55042bfbf741c38" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:14:36.096834 26558 raft_consensus.cc:2454] T 2924b2da6f8347aca55042bfbf741c38 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:14:36.096931 26817 raft_consensus.cc:2454] T 2924b2da6f8347aca55042bfbf741c38 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:14:36.097398 26625 leader_election.cc:302] T 2924b2da6f8347aca55042bfbf741c38 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:36.097693 28279 raft_consensus.cc:2781] T 2924b2da6f8347aca55042bfbf741c38 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:36.097766 28279 raft_consensus.cc:482] T 2924b2da6f8347aca55042bfbf741c38 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:36.097848 28279 raft_consensus.cc:3037] T 2924b2da6f8347aca55042bfbf741c38 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:36.100209 28279 raft_consensus.cc:504] T 2924b2da6f8347aca55042bfbf741c38 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:36.100672 28435 tablet_bootstrap.cc:492] T 1a147a6c2ba04d848a122021fd7c63c6 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:36.100805 28279 leader_election.cc:288] T 2924b2da6f8347aca55042bfbf741c38 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Requested vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:36.100852 28435 ts_tablet_manager.cc:1367] T 1a147a6c2ba04d848a122021fd7c63c6 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.000s
I20240502 14:14:36.101316 26817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2924b2da6f8347aca55042bfbf741c38" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:14:36.101449 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2924b2da6f8347aca55042bfbf741c38" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:14:36.101572 26817 raft_consensus.cc:3037] T 2924b2da6f8347aca55042bfbf741c38 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:36.101706 26558 raft_consensus.cc:3037] T 2924b2da6f8347aca55042bfbf741c38 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:36.101734 28435 raft_consensus.cc:348] T 1a147a6c2ba04d848a122021fd7c63c6 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:36.101982 28435 raft_consensus.cc:374] T 1a147a6c2ba04d848a122021fd7c63c6 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:36.102087 28435 raft_consensus.cc:729] T 1a147a6c2ba04d848a122021fd7c63c6 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:36.102398 28435 consensus_queue.cc:260] T 1a147a6c2ba04d848a122021fd7c63c6 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:36.103488 28435 ts_tablet_manager.cc:1398] T 1a147a6c2ba04d848a122021fd7c63c6 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:36.104470 26558 raft_consensus.cc:2454] T 2924b2da6f8347aca55042bfbf741c38 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:14:36.104903 26817 raft_consensus.cc:2454] T 2924b2da6f8347aca55042bfbf741c38 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:14:36.104911 26624 leader_election.cc:302] T 2924b2da6f8347aca55042bfbf741c38 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:36.105300 28279 raft_consensus.cc:2781] T 2924b2da6f8347aca55042bfbf741c38 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:36.105509 28279 raft_consensus.cc:686] T 2924b2da6f8347aca55042bfbf741c38 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Becoming Leader. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Running, Role: LEADER
I20240502 14:14:36.105765 28279 consensus_queue.cc:237] T 2924b2da6f8347aca55042bfbf741c38 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:36.109601 26282 catalog_manager.cc:5484] T 2924b2da6f8347aca55042bfbf741c38 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195). New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:14:36.185768 28279 raft_consensus.cc:482] T e79bc37363cc48e7943e0275981be31a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:36.185961 28279 raft_consensus.cc:504] T e79bc37363cc48e7943e0275981be31a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:36.186575 28279 leader_election.cc:288] T e79bc37363cc48e7943e0275981be31a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:14:36.187119 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e79bc37363cc48e7943e0275981be31a" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:36.187144 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e79bc37363cc48e7943e0275981be31a" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:14:36.187352 26558 raft_consensus.cc:2454] T e79bc37363cc48e7943e0275981be31a P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:14:36.187359 26426 raft_consensus.cc:2454] T e79bc37363cc48e7943e0275981be31a P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:14:36.187803 26624 leader_election.cc:302] T e79bc37363cc48e7943e0275981be31a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:14:36.188087 28279 raft_consensus.cc:2781] T e79bc37363cc48e7943e0275981be31a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:36.188220 28279 raft_consensus.cc:482] T e79bc37363cc48e7943e0275981be31a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:36.188330 28279 raft_consensus.cc:3037] T e79bc37363cc48e7943e0275981be31a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:36.190351 28279 raft_consensus.cc:504] T e79bc37363cc48e7943e0275981be31a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:36.190958 28279 leader_election.cc:288] T e79bc37363cc48e7943e0275981be31a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Requested vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:14:36.191566 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e79bc37363cc48e7943e0275981be31a" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:14:36.191694 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e79bc37363cc48e7943e0275981be31a" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:14:36.191836 26558 raft_consensus.cc:3037] T e79bc37363cc48e7943e0275981be31a P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:36.191927 26426 raft_consensus.cc:3037] T e79bc37363cc48e7943e0275981be31a P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:36.194873 26558 raft_consensus.cc:2454] T e79bc37363cc48e7943e0275981be31a P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:14:36.194875 26426 raft_consensus.cc:2454] T e79bc37363cc48e7943e0275981be31a P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:14:36.195344 26624 leader_election.cc:302] T e79bc37363cc48e7943e0275981be31a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:14:36.195645 28279 raft_consensus.cc:2781] T e79bc37363cc48e7943e0275981be31a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:36.195947 28279 raft_consensus.cc:686] T e79bc37363cc48e7943e0275981be31a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Becoming Leader. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Running, Role: LEADER
I20240502 14:14:36.196254 28279 consensus_queue.cc:237] T e79bc37363cc48e7943e0275981be31a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:36.199589 26282 catalog_manager.cc:5484] T e79bc37363cc48e7943e0275981be31a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195). New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:14:36.230257 28026 raft_consensus.cc:482] T 1a147a6c2ba04d848a122021fd7c63c6 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:36.230515 28026 raft_consensus.cc:504] T 1a147a6c2ba04d848a122021fd7c63c6 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:36.231268 28026 leader_election.cc:288] T 1a147a6c2ba04d848a122021fd7c63c6 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:36.231871 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a147a6c2ba04d848a122021fd7c63c6" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:14:36.231987 26817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a147a6c2ba04d848a122021fd7c63c6" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:14:36.232177 26426 raft_consensus.cc:2454] T 1a147a6c2ba04d848a122021fd7c63c6 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 0.
I20240502 14:14:36.232187 26817 raft_consensus.cc:2454] T 1a147a6c2ba04d848a122021fd7c63c6 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 0.
I20240502 14:14:36.232684 26494 leader_election.cc:302] T 1a147a6c2ba04d848a122021fd7c63c6 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:36.233004 28026 raft_consensus.cc:2781] T 1a147a6c2ba04d848a122021fd7c63c6 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:36.233106 28026 raft_consensus.cc:482] T 1a147a6c2ba04d848a122021fd7c63c6 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:36.233233 28026 raft_consensus.cc:3037] T 1a147a6c2ba04d848a122021fd7c63c6 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:36.237159 28026 raft_consensus.cc:504] T 1a147a6c2ba04d848a122021fd7c63c6 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:36.238143 28026 leader_election.cc:288] T 1a147a6c2ba04d848a122021fd7c63c6 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Requested vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:36.238610 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a147a6c2ba04d848a122021fd7c63c6" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:14:36.238879 26426 raft_consensus.cc:3037] T 1a147a6c2ba04d848a122021fd7c63c6 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:36.241022 26426 raft_consensus.cc:2454] T 1a147a6c2ba04d848a122021fd7c63c6 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:14:36.241434 26817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a147a6c2ba04d848a122021fd7c63c6" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:14:36.241482 26493 leader_election.cc:302] T 1a147a6c2ba04d848a122021fd7c63c6 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:36.241648 26817 raft_consensus.cc:3037] T 1a147a6c2ba04d848a122021fd7c63c6 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:36.246342 26817 raft_consensus.cc:2454] T 1a147a6c2ba04d848a122021fd7c63c6 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:14:36.247385 28026 raft_consensus.cc:2781] T 1a147a6c2ba04d848a122021fd7c63c6 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:36.247793 28026 raft_consensus.cc:686] T 1a147a6c2ba04d848a122021fd7c63c6 P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Becoming Leader. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Running, Role: LEADER
I20240502 14:14:36.248133 28026 consensus_queue.cc:237] T 1a147a6c2ba04d848a122021fd7c63c6 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:36.259407 26282 catalog_manager.cc:5484] T 1a147a6c2ba04d848a122021fd7c63c6 P ba187e38a2754d60a12de9ac4856430a reported cstate change: term changed from 0 to 1, leader changed from <none> to ba187e38a2754d60a12de9ac4856430a (127.19.94.194). New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:14:36.592069 28279 consensus_queue.cc:1035] T 2924b2da6f8347aca55042bfbf741c38 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:36.599179 28279 consensus_queue.cc:1035] T 2924b2da6f8347aca55042bfbf741c38 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:36.627943 28445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:36.628271 28445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 14:14:36.632288 27953 consensus_queue.cc:1035] T e79bc37363cc48e7943e0275981be31a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:36.637950 28445 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:14:36.638092 27953 consensus_queue.cc:1035] T e79bc37363cc48e7943e0275981be31a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_12'
I20240502 14:14:36.696527 28026 consensus_queue.cc:1035] T 1a147a6c2ba04d848a122021fd7c63c6 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:36.700765 28026 consensus_queue.cc:1035] T 1a147a6c2ba04d848a122021fd7c63c6 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:36.868139 26624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.193:37985, user_credentials={real_user=slave}} blocked reactor thread for 45167.9us
W20240502 14:14:36.905519 26625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.196:32843, user_credentials={real_user=slave}} blocked reactor thread for 65691.6us
INSERT report
rows total: 2000
time total: 285.066 ms
time per row: 0.142533 ms
I20240502 14:14:37.107609 26278 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalance_test_table_13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:37.117116 26406 tablet_service.cc:1467] Processing CreateTablet for tablet c41522c60f3942aa9f28f514dcad4e49 (DEFAULT_TABLE table=rebalance_test_table_13 [id=835ba73992f74008a1a10f4c18f2db38]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:37.117666 26406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c41522c60f3942aa9f28f514dcad4e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:37.117583 26667 tablet_service.cc:1467] Processing CreateTablet for tablet c41522c60f3942aa9f28f514dcad4e49 (DEFAULT_TABLE table=rebalance_test_table_13 [id=835ba73992f74008a1a10f4c18f2db38]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:37.118039 26667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c41522c60f3942aa9f28f514dcad4e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:37.118606 26799 tablet_service.cc:1467] Processing CreateTablet for tablet c41522c60f3942aa9f28f514dcad4e49 (DEFAULT_TABLE table=rebalance_test_table_13 [id=835ba73992f74008a1a10f4c18f2db38]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:37.119025 26799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c41522c60f3942aa9f28f514dcad4e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:37.120043 26537 tablet_service.cc:1467] Processing CreateTablet for tablet 6814613d36a74353a99673cf16aa9821 (DEFAULT_TABLE table=rebalance_test_table_13 [id=835ba73992f74008a1a10f4c18f2db38]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:37.120525 26537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6814613d36a74353a99673cf16aa9821. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:37.124975 26669 tablet_service.cc:1467] Processing CreateTablet for tablet 1fb1ef75829641a89c4cac539f431d80 (DEFAULT_TABLE table=rebalance_test_table_13 [id=835ba73992f74008a1a10f4c18f2db38]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:37.125200 26798 tablet_service.cc:1467] Processing CreateTablet for tablet 1fb1ef75829641a89c4cac539f431d80 (DEFAULT_TABLE table=rebalance_test_table_13 [id=835ba73992f74008a1a10f4c18f2db38]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:37.125418 26669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fb1ef75829641a89c4cac539f431d80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:37.125655 26798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fb1ef75829641a89c4cac539f431d80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:37.128803 26405 tablet_service.cc:1467] Processing CreateTablet for tablet 6814613d36a74353a99673cf16aa9821 (DEFAULT_TABLE table=rebalance_test_table_13 [id=835ba73992f74008a1a10f4c18f2db38]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:37.129204 26405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6814613d36a74353a99673cf16aa9821. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:37.133018 26407 tablet_service.cc:1467] Processing CreateTablet for tablet 1fb1ef75829641a89c4cac539f431d80 (DEFAULT_TABLE table=rebalance_test_table_13 [id=835ba73992f74008a1a10f4c18f2db38]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:37.133558 26407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fb1ef75829641a89c4cac539f431d80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:37.125173 26797 tablet_service.cc:1467] Processing CreateTablet for tablet 6814613d36a74353a99673cf16aa9821 (DEFAULT_TABLE table=rebalance_test_table_13 [id=835ba73992f74008a1a10f4c18f2db38]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:37.137120 26797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6814613d36a74353a99673cf16aa9821. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:37.141381 28514 tablet_bootstrap.cc:492] T c41522c60f3942aa9f28f514dcad4e49 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:37.146060 28515 tablet_bootstrap.cc:492] T 6814613d36a74353a99673cf16aa9821 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:37.148058 28514 tablet_bootstrap.cc:654] T c41522c60f3942aa9f28f514dcad4e49 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:37.149107 28515 tablet_bootstrap.cc:654] T 6814613d36a74353a99673cf16aa9821 P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:37.149703 28516 tablet_bootstrap.cc:492] T c41522c60f3942aa9f28f514dcad4e49 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:37.149969 28517 tablet_bootstrap.cc:492] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:37.152001 28517 tablet_bootstrap.cc:654] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:37.155045 28516 tablet_bootstrap.cc:654] T c41522c60f3942aa9f28f514dcad4e49 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:37.156574 28517 tablet_bootstrap.cc:492] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:37.156801 28517 ts_tablet_manager.cc:1367] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.002s
I20240502 14:14:37.157758 28517 raft_consensus.cc:348] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:37.158074 28517 raft_consensus.cc:374] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:37.158229 28517 raft_consensus.cc:729] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:37.158663 28517 consensus_queue.cc:260] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:37.159644 28517 ts_tablet_manager.cc:1398] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:37.160002 28517 tablet_bootstrap.cc:492] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:37.161652 28517 tablet_bootstrap.cc:654] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:37.163367 28515 tablet_bootstrap.cc:492] T 6814613d36a74353a99673cf16aa9821 P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:37.163398 28514 tablet_bootstrap.cc:492] T c41522c60f3942aa9f28f514dcad4e49 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:37.163547 28514 ts_tablet_manager.cc:1367] T c41522c60f3942aa9f28f514dcad4e49 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.011s
I20240502 14:14:37.163842 28515 ts_tablet_manager.cc:1367] T 6814613d36a74353a99673cf16aa9821 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.012s
I20240502 14:14:37.164521 28514 raft_consensus.cc:348] T c41522c60f3942aa9f28f514dcad4e49 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:37.164700 28515 raft_consensus.cc:348] T 6814613d36a74353a99673cf16aa9821 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:37.164829 28514 raft_consensus.cc:374] T c41522c60f3942aa9f28f514dcad4e49 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:37.164925 28514 raft_consensus.cc:729] T c41522c60f3942aa9f28f514dcad4e49 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:37.164973 28515 raft_consensus.cc:374] T 6814613d36a74353a99673cf16aa9821 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:37.165115 28515 raft_consensus.cc:729] T 6814613d36a74353a99673cf16aa9821 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:37.165257 28514 consensus_queue.cc:260] T c41522c60f3942aa9f28f514dcad4e49 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:37.165436 28515 consensus_queue.cc:260] T 6814613d36a74353a99673cf16aa9821 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:37.165975 28514 ts_tablet_manager.cc:1398] T c41522c60f3942aa9f28f514dcad4e49 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:14:37.166205 28515 ts_tablet_manager.cc:1398] T 6814613d36a74353a99673cf16aa9821 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:37.167579 28514 tablet_bootstrap.cc:492] T 6814613d36a74353a99673cf16aa9821 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:37.169925 28514 tablet_bootstrap.cc:654] T 6814613d36a74353a99673cf16aa9821 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:37.173123 28517 tablet_bootstrap.cc:492] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:37.173373 28517 ts_tablet_manager.cc:1367] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20240502 14:14:37.173712 28516 tablet_bootstrap.cc:492] T c41522c60f3942aa9f28f514dcad4e49 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:37.173882 28516 ts_tablet_manager.cc:1367] T c41522c60f3942aa9f28f514dcad4e49 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.006s
I20240502 14:14:37.174362 28517 raft_consensus.cc:348] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:37.174688 28517 raft_consensus.cc:374] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:37.174840 28517 raft_consensus.cc:729] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:37.174789 28516 raft_consensus.cc:348] T c41522c60f3942aa9f28f514dcad4e49 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:37.175071 28516 raft_consensus.cc:374] T c41522c60f3942aa9f28f514dcad4e49 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:37.175195 28516 raft_consensus.cc:729] T c41522c60f3942aa9f28f514dcad4e49 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:37.175276 28517 consensus_queue.cc:260] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:37.175491 28516 consensus_queue.cc:260] T c41522c60f3942aa9f28f514dcad4e49 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:37.176237 28517 ts_tablet_manager.cc:1398] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:14:37.176268 28516 ts_tablet_manager.cc:1398] T c41522c60f3942aa9f28f514dcad4e49 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:14:37.176653 28516 tablet_bootstrap.cc:492] T 1fb1ef75829641a89c4cac539f431d80 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:37.179196 28516 tablet_bootstrap.cc:654] T 1fb1ef75829641a89c4cac539f431d80 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:37.180284 28514 tablet_bootstrap.cc:492] T 6814613d36a74353a99673cf16aa9821 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:37.180531 28514 ts_tablet_manager.cc:1367] T 6814613d36a74353a99673cf16aa9821 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20240502 14:14:37.181401 28514 raft_consensus.cc:348] T 6814613d36a74353a99673cf16aa9821 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:37.181676 28514 raft_consensus.cc:374] T 6814613d36a74353a99673cf16aa9821 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:37.181802 28514 raft_consensus.cc:729] T 6814613d36a74353a99673cf16aa9821 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:37.182392 28514 consensus_queue.cc:260] T 6814613d36a74353a99673cf16aa9821 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:37.183024 28514 ts_tablet_manager.cc:1398] T 6814613d36a74353a99673cf16aa9821 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:37.183395 28516 tablet_bootstrap.cc:492] T 1fb1ef75829641a89c4cac539f431d80 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:37.183425 28514 tablet_bootstrap.cc:492] T 1fb1ef75829641a89c4cac539f431d80 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:37.183566 28516 ts_tablet_manager.cc:1367] T 1fb1ef75829641a89c4cac539f431d80 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20240502 14:14:37.184351 28516 raft_consensus.cc:348] T 1fb1ef75829641a89c4cac539f431d80 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:37.184602 28516 raft_consensus.cc:374] T 1fb1ef75829641a89c4cac539f431d80 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:37.184686 28516 raft_consensus.cc:729] T 1fb1ef75829641a89c4cac539f431d80 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:37.185078 28514 tablet_bootstrap.cc:654] T 1fb1ef75829641a89c4cac539f431d80 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:37.184947 28516 consensus_queue.cc:260] T 1fb1ef75829641a89c4cac539f431d80 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:37.185555 28516 ts_tablet_manager.cc:1398] T 1fb1ef75829641a89c4cac539f431d80 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:37.186010 28516 tablet_bootstrap.cc:492] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:37.188380 28516 tablet_bootstrap.cc:654] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:37.190222 28514 tablet_bootstrap.cc:492] T 1fb1ef75829641a89c4cac539f431d80 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:37.190450 28514 ts_tablet_manager.cc:1367] T 1fb1ef75829641a89c4cac539f431d80 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.002s
I20240502 14:14:37.191243 28516 tablet_bootstrap.cc:492] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:37.191422 28516 ts_tablet_manager.cc:1367] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 14:14:37.191449 28514 raft_consensus.cc:348] T 1fb1ef75829641a89c4cac539f431d80 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:37.191679 28514 raft_consensus.cc:374] T 1fb1ef75829641a89c4cac539f431d80 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:37.191777 28514 raft_consensus.cc:729] T 1fb1ef75829641a89c4cac539f431d80 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:37.192014 28514 consensus_queue.cc:260] T 1fb1ef75829641a89c4cac539f431d80 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:37.192193 28516 raft_consensus.cc:348] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:37.192436 28516 raft_consensus.cc:374] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:37.192545 28516 raft_consensus.cc:729] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:37.192598 28514 ts_tablet_manager.cc:1398] T 1fb1ef75829641a89c4cac539f431d80 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:37.192801 28516 consensus_queue.cc:260] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:37.193403 28516 ts_tablet_manager.cc:1398] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:37.268486 28486 raft_consensus.cc:482] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:37.268715 28486 raft_consensus.cc:504] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:37.269506 28279 raft_consensus.cc:482] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:37.269774 28486 leader_election.cc:288] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:37.269699 28279 raft_consensus.cc:504] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:37.270390 28279 leader_election.cc:288] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:14:37.270396 26425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c41522c60f3942aa9f28f514dcad4e49" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:14:37.270644 26425 raft_consensus.cc:2454] T c41522c60f3942aa9f28f514dcad4e49 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:14:37.271026 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fb1ef75829641a89c4cac539f431d80" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:14:37.271065 26817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c41522c60f3942aa9f28f514dcad4e49" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:14:37.271270 26624 leader_election.cc:302] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:14:37.271306 26818 raft_consensus.cc:2454] T 1fb1ef75829641a89c4cac539f431d80 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:14:37.271368 26817 raft_consensus.cc:2454] T c41522c60f3942aa9f28f514dcad4e49 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:14:37.271601 28279 raft_consensus.cc:2781] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:37.271706 28279 raft_consensus.cc:482] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:37.271747 26625 leader_election.cc:302] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:37.271675 26425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fb1ef75829641a89c4cac539f431d80" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:14:37.271836 28279 raft_consensus.cc:3037] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:37.271884 26425 raft_consensus.cc:2454] T 1fb1ef75829641a89c4cac539f431d80 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:14:37.272042 28486 raft_consensus.cc:2781] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:37.272168 28486 raft_consensus.cc:482] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:37.272289 28486 raft_consensus.cc:3037] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:37.274204 28279 raft_consensus.cc:504] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:37.274281 28486 raft_consensus.cc:504] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:37.274928 28279 leader_election.cc:288] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Requested vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:37.274935 28486 leader_election.cc:288] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Requested vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:14:37.275595 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fb1ef75829641a89c4cac539f431d80" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:14:37.275696 26425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c41522c60f3942aa9f28f514dcad4e49" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:14:37.275748 26817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c41522c60f3942aa9f28f514dcad4e49" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:14:37.275785 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fb1ef75829641a89c4cac539f431d80" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:14:37.275867 26818 raft_consensus.cc:3037] T 1fb1ef75829641a89c4cac539f431d80 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:37.275938 26425 raft_consensus.cc:3037] T c41522c60f3942aa9f28f514dcad4e49 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:37.275964 26817 raft_consensus.cc:3037] T c41522c60f3942aa9f28f514dcad4e49 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:37.275990 26426 raft_consensus.cc:3037] T 1fb1ef75829641a89c4cac539f431d80 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:37.279062 26425 raft_consensus.cc:2454] T c41522c60f3942aa9f28f514dcad4e49 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:14:37.279111 26818 raft_consensus.cc:2454] T 1fb1ef75829641a89c4cac539f431d80 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:14:37.279138 26426 raft_consensus.cc:2454] T 1fb1ef75829641a89c4cac539f431d80 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:14:37.279191 26817 raft_consensus.cc:2454] T c41522c60f3942aa9f28f514dcad4e49 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:14:37.279619 26625 leader_election.cc:302] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:37.279580 26624 leader_election.cc:302] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:14:37.279995 28279 raft_consensus.cc:2781] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:37.279995 28486 raft_consensus.cc:2781] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:37.280203 28486 raft_consensus.cc:686] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Becoming Leader. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Running, Role: LEADER
I20240502 14:14:37.280297 28279 raft_consensus.cc:686] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Becoming Leader. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Running, Role: LEADER
I20240502 14:14:37.280584 28279 consensus_queue.cc:237] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:37.280607 28486 consensus_queue.cc:237] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:37.285374 26277 catalog_manager.cc:5484] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195). New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:14:37.285780 26277 catalog_manager.cc:5484] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195). New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:14:37.472095 28176 raft_consensus.cc:482] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:37.472326 28176 raft_consensus.cc:504] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:37.473280 28176 leader_election.cc:288] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:37.473693 26556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6814613d36a74353a99673cf16aa9821" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:37.473768 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6814613d36a74353a99673cf16aa9821" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:14:37.473897 26556 raft_consensus.cc:2454] T 6814613d36a74353a99673cf16aa9821 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:14:37.474013 26818 raft_consensus.cc:2454] T 6814613d36a74353a99673cf16aa9821 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:14:37.474349 26362 leader_election.cc:302] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:37.474658 28176 raft_consensus.cc:2781] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:37.474757 28176 raft_consensus.cc:482] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:37.474858 28176 raft_consensus.cc:3037] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:37.476835 28176 raft_consensus.cc:504] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:37.477380 28176 leader_election.cc:288] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Requested vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:37.477941 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6814613d36a74353a99673cf16aa9821" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:14:37.477941 26556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6814613d36a74353a99673cf16aa9821" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:14:37.478154 26556 raft_consensus.cc:3037] T 6814613d36a74353a99673cf16aa9821 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:37.478154 26818 raft_consensus.cc:3037] T 6814613d36a74353a99673cf16aa9821 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:37.481024 26556 raft_consensus.cc:2454] T 6814613d36a74353a99673cf16aa9821 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:14:37.481024 26818 raft_consensus.cc:2454] T 6814613d36a74353a99673cf16aa9821 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:14:37.481477 26362 leader_election.cc:302] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:37.481793 28176 raft_consensus.cc:2781] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:37.481937 28176 raft_consensus.cc:686] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Becoming Leader. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Running, Role: LEADER
I20240502 14:14:37.482239 28176 consensus_queue.cc:237] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:37.485270 26277 catalog_manager.cc:5484] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: term changed from 0 to 1, leader changed from <none> to 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193). New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:14:37.662357 28279 consensus_queue.cc:1035] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:37.679198 28486 consensus_queue.cc:1035] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:37.720041 28486 consensus_queue.cc:1035] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:37.725466 27953 consensus_queue.cc:1035] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:37.893066 28531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:37.893456 28531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:37.901954 28531 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_13'
I20240502 14:14:37.990376 28529 consensus_queue.cc:1035] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:38.002636 28529 consensus_queue.cc:1035] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:38.274421 26362 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.194:36025, user_credentials={real_user=slave}} blocked reactor thread for 60360us
INSERT report
rows total: 2000
time total: 330.673 ms
time per row: 0.165337 ms
I20240502 14:14:38.397567 26279 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalance_test_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:38.408083 26537 tablet_service.cc:1467] Processing CreateTablet for tablet 6f3334b24a3145e0b5498d8d97eabb1f (DEFAULT_TABLE table=rebalance_test_table_14 [id=4b7658f1ae4f4326bedadc9a5b8d88c2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:38.408480 26667 tablet_service.cc:1467] Processing CreateTablet for tablet 6f3334b24a3145e0b5498d8d97eabb1f (DEFAULT_TABLE table=rebalance_test_table_14 [id=4b7658f1ae4f4326bedadc9a5b8d88c2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:38.408682 26537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f3334b24a3145e0b5498d8d97eabb1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:38.408972 26667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f3334b24a3145e0b5498d8d97eabb1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:38.410581 26538 tablet_service.cc:1467] Processing CreateTablet for tablet f67582dfce354a9b92078ab0b5799194 (DEFAULT_TABLE table=rebalance_test_table_14 [id=4b7658f1ae4f4326bedadc9a5b8d88c2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:38.410998 26538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f67582dfce354a9b92078ab0b5799194. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:38.411345 26668 tablet_service.cc:1467] Processing CreateTablet for tablet 038a5d3d5368433cac07bdd94a4f779a (DEFAULT_TABLE table=rebalance_test_table_14 [id=4b7658f1ae4f4326bedadc9a5b8d88c2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:38.411775 26668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 038a5d3d5368433cac07bdd94a4f779a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:38.417037 26536 tablet_service.cc:1467] Processing CreateTablet for tablet 038a5d3d5368433cac07bdd94a4f779a (DEFAULT_TABLE table=rebalance_test_table_14 [id=4b7658f1ae4f4326bedadc9a5b8d88c2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:38.417639 26536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 038a5d3d5368433cac07bdd94a4f779a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:38.420007 28599 tablet_bootstrap.cc:492] T 6f3334b24a3145e0b5498d8d97eabb1f P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:38.421082 26669 tablet_service.cc:1467] Processing CreateTablet for tablet f67582dfce354a9b92078ab0b5799194 (DEFAULT_TABLE table=rebalance_test_table_14 [id=4b7658f1ae4f4326bedadc9a5b8d88c2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:38.422080 28599 tablet_bootstrap.cc:654] T 6f3334b24a3145e0b5498d8d97eabb1f P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:38.423424 26405 tablet_service.cc:1467] Processing CreateTablet for tablet 038a5d3d5368433cac07bdd94a4f779a (DEFAULT_TABLE table=rebalance_test_table_14 [id=4b7658f1ae4f4326bedadc9a5b8d88c2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:38.424062 26669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f67582dfce354a9b92078ab0b5799194. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:38.424073 26405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 038a5d3d5368433cac07bdd94a4f779a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:38.429519 26406 tablet_service.cc:1467] Processing CreateTablet for tablet 6f3334b24a3145e0b5498d8d97eabb1f (DEFAULT_TABLE table=rebalance_test_table_14 [id=4b7658f1ae4f4326bedadc9a5b8d88c2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:38.429724 26798 tablet_service.cc:1467] Processing CreateTablet for tablet f67582dfce354a9b92078ab0b5799194 (DEFAULT_TABLE table=rebalance_test_table_14 [id=4b7658f1ae4f4326bedadc9a5b8d88c2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:38.430097 26406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f3334b24a3145e0b5498d8d97eabb1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:38.430330 26798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f67582dfce354a9b92078ab0b5799194. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:38.456959 28601 tablet_bootstrap.cc:492] T f67582dfce354a9b92078ab0b5799194 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:38.458225 28598 tablet_bootstrap.cc:492] T 6f3334b24a3145e0b5498d8d97eabb1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:38.459591 28602 tablet_bootstrap.cc:492] T 6f3334b24a3145e0b5498d8d97eabb1f P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:38.461675 28598 tablet_bootstrap.cc:654] T 6f3334b24a3145e0b5498d8d97eabb1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:38.461970 28602 tablet_bootstrap.cc:654] T 6f3334b24a3145e0b5498d8d97eabb1f P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:38.464056 28601 tablet_bootstrap.cc:654] T f67582dfce354a9b92078ab0b5799194 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:38.464468 28602 tablet_bootstrap.cc:492] T 6f3334b24a3145e0b5498d8d97eabb1f P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:38.464701 28602 ts_tablet_manager.cc:1367] T 6f3334b24a3145e0b5498d8d97eabb1f P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240502 14:14:38.465050 28599 tablet_bootstrap.cc:492] T 6f3334b24a3145e0b5498d8d97eabb1f P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:38.465272 28599 ts_tablet_manager.cc:1367] T 6f3334b24a3145e0b5498d8d97eabb1f P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.045s user 0.021s sys 0.017s
I20240502 14:14:38.465770 28602 raft_consensus.cc:348] T 6f3334b24a3145e0b5498d8d97eabb1f P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:38.466156 28602 raft_consensus.cc:374] T 6f3334b24a3145e0b5498d8d97eabb1f P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:38.466043 28599 raft_consensus.cc:348] T 6f3334b24a3145e0b5498d8d97eabb1f P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:38.466271 28599 raft_consensus.cc:374] T 6f3334b24a3145e0b5498d8d97eabb1f P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:38.466277 28602 raft_consensus.cc:729] T 6f3334b24a3145e0b5498d8d97eabb1f P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:38.466384 28599 raft_consensus.cc:729] T 6f3334b24a3145e0b5498d8d97eabb1f P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:38.466532 28602 consensus_queue.cc:260] T 6f3334b24a3145e0b5498d8d97eabb1f P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:38.466660 28599 consensus_queue.cc:260] T 6f3334b24a3145e0b5498d8d97eabb1f P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:38.467264 28602 ts_tablet_manager.cc:1398] T 6f3334b24a3145e0b5498d8d97eabb1f P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:38.467373 28599 ts_tablet_manager.cc:1398] T 6f3334b24a3145e0b5498d8d97eabb1f P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:38.467597 28602 tablet_bootstrap.cc:492] T 038a5d3d5368433cac07bdd94a4f779a P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:38.467604 28598 tablet_bootstrap.cc:492] T 6f3334b24a3145e0b5498d8d97eabb1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:38.467759 28598 ts_tablet_manager.cc:1367] T 6f3334b24a3145e0b5498d8d97eabb1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20240502 14:14:38.468139 28601 tablet_bootstrap.cc:492] T f67582dfce354a9b92078ab0b5799194 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:38.468290 28601 ts_tablet_manager.cc:1367] T f67582dfce354a9b92078ab0b5799194 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.005s
I20240502 14:14:38.468622 28598 raft_consensus.cc:348] T 6f3334b24a3145e0b5498d8d97eabb1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:38.469067 28599 tablet_bootstrap.cc:492] T f67582dfce354a9b92078ab0b5799194 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:38.469326 28602 tablet_bootstrap.cc:654] T 038a5d3d5368433cac07bdd94a4f779a P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:38.470818 28599 tablet_bootstrap.cc:654] T f67582dfce354a9b92078ab0b5799194 P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:38.471096 28602 tablet_bootstrap.cc:492] T 038a5d3d5368433cac07bdd94a4f779a P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:38.471577 28602 ts_tablet_manager.cc:1367] T 038a5d3d5368433cac07bdd94a4f779a P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:14:38.471824 28598 raft_consensus.cc:374] T 6f3334b24a3145e0b5498d8d97eabb1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:38.471989 28598 raft_consensus.cc:729] T 6f3334b24a3145e0b5498d8d97eabb1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:38.472290 28598 consensus_queue.cc:260] T 6f3334b24a3145e0b5498d8d97eabb1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:38.472442 28602 raft_consensus.cc:348] T 038a5d3d5368433cac07bdd94a4f779a P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:38.472680 28602 raft_consensus.cc:374] T 038a5d3d5368433cac07bdd94a4f779a P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:38.472770 28602 raft_consensus.cc:729] T 038a5d3d5368433cac07bdd94a4f779a P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:38.473074 28602 consensus_queue.cc:260] T 038a5d3d5368433cac07bdd94a4f779a P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:38.473201 28598 ts_tablet_manager.cc:1398] T 6f3334b24a3145e0b5498d8d97eabb1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:14:38.473752 28602 ts_tablet_manager.cc:1398] T 038a5d3d5368433cac07bdd94a4f779a P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:38.474750 28598 tablet_bootstrap.cc:492] T 038a5d3d5368433cac07bdd94a4f779a P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:38.476799 28598 tablet_bootstrap.cc:654] T 038a5d3d5368433cac07bdd94a4f779a P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:38.474225 28601 raft_consensus.cc:348] T f67582dfce354a9b92078ab0b5799194 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:38.478121 28601 raft_consensus.cc:374] T f67582dfce354a9b92078ab0b5799194 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:38.478232 28601 raft_consensus.cc:729] T f67582dfce354a9b92078ab0b5799194 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:38.478514 28601 consensus_queue.cc:260] T f67582dfce354a9b92078ab0b5799194 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:38.479282 28601 ts_tablet_manager.cc:1398] T f67582dfce354a9b92078ab0b5799194 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 14:14:38.481233 28599 tablet_bootstrap.cc:492] T f67582dfce354a9b92078ab0b5799194 P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:38.481434 28599 ts_tablet_manager.cc:1367] T f67582dfce354a9b92078ab0b5799194 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.001s
I20240502 14:14:38.482295 28599 raft_consensus.cc:348] T f67582dfce354a9b92078ab0b5799194 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:38.482553 28599 raft_consensus.cc:374] T f67582dfce354a9b92078ab0b5799194 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:38.482640 28599 raft_consensus.cc:729] T f67582dfce354a9b92078ab0b5799194 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:38.482877 28599 consensus_queue.cc:260] T f67582dfce354a9b92078ab0b5799194 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:38.483431 28599 ts_tablet_manager.cc:1398] T f67582dfce354a9b92078ab0b5799194 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:14:38.484046 28599 tablet_bootstrap.cc:492] T 038a5d3d5368433cac07bdd94a4f779a P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:38.484526 28598 tablet_bootstrap.cc:492] T 038a5d3d5368433cac07bdd94a4f779a P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:38.484709 28598 ts_tablet_manager.cc:1367] T 038a5d3d5368433cac07bdd94a4f779a P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.001s
I20240502 14:14:38.485436 28598 raft_consensus.cc:348] T 038a5d3d5368433cac07bdd94a4f779a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:38.485723 28598 raft_consensus.cc:374] T 038a5d3d5368433cac07bdd94a4f779a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:38.485852 28598 raft_consensus.cc:729] T 038a5d3d5368433cac07bdd94a4f779a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:38.486181 28598 consensus_queue.cc:260] T 038a5d3d5368433cac07bdd94a4f779a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:38.487154 28599 tablet_bootstrap.cc:654] T 038a5d3d5368433cac07bdd94a4f779a P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:38.487859 28598 ts_tablet_manager.cc:1398] T 038a5d3d5368433cac07bdd94a4f779a P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:14:38.488201 28598 tablet_bootstrap.cc:492] T f67582dfce354a9b92078ab0b5799194 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:38.490631 28598 tablet_bootstrap.cc:654] T f67582dfce354a9b92078ab0b5799194 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:38.492383 28599 tablet_bootstrap.cc:492] T 038a5d3d5368433cac07bdd94a4f779a P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:38.492659 28599 ts_tablet_manager.cc:1367] T 038a5d3d5368433cac07bdd94a4f779a P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.002s
I20240502 14:14:38.493127 28598 tablet_bootstrap.cc:492] T f67582dfce354a9b92078ab0b5799194 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:38.493285 28598 ts_tablet_manager.cc:1367] T f67582dfce354a9b92078ab0b5799194 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 14:14:38.493417 28599 raft_consensus.cc:348] T 038a5d3d5368433cac07bdd94a4f779a P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:38.493605 28599 raft_consensus.cc:374] T 038a5d3d5368433cac07bdd94a4f779a P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:38.493702 28599 raft_consensus.cc:729] T 038a5d3d5368433cac07bdd94a4f779a P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:38.493939 28599 consensus_queue.cc:260] T 038a5d3d5368433cac07bdd94a4f779a P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:38.494072 28598 raft_consensus.cc:348] T f67582dfce354a9b92078ab0b5799194 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:38.494285 28598 raft_consensus.cc:374] T f67582dfce354a9b92078ab0b5799194 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:38.494390 28598 raft_consensus.cc:729] T f67582dfce354a9b92078ab0b5799194 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:38.494534 28599 ts_tablet_manager.cc:1398] T 038a5d3d5368433cac07bdd94a4f779a P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:14:38.494625 28598 consensus_queue.cc:260] T f67582dfce354a9b92078ab0b5799194 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:38.495296 28598 ts_tablet_manager.cc:1398] T f67582dfce354a9b92078ab0b5799194 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:38.541198 28529 raft_consensus.cc:482] T 038a5d3d5368433cac07bdd94a4f779a P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:38.541389 28529 raft_consensus.cc:504] T 038a5d3d5368433cac07bdd94a4f779a P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:38.542292 28529 leader_election.cc:288] T 038a5d3d5368433cac07bdd94a4f779a P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:14:38.542858 26689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "038a5d3d5368433cac07bdd94a4f779a" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:14:38.542865 26556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "038a5d3d5368433cac07bdd94a4f779a" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:38.543123 26689 raft_consensus.cc:2454] T 038a5d3d5368433cac07bdd94a4f779a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:14:38.543118 26556 raft_consensus.cc:2454] T 038a5d3d5368433cac07bdd94a4f779a P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:14:38.543629 26362 leader_election.cc:302] T 038a5d3d5368433cac07bdd94a4f779a P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:38.543900 28529 raft_consensus.cc:2781] T 038a5d3d5368433cac07bdd94a4f779a P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:38.544024 28529 raft_consensus.cc:482] T 038a5d3d5368433cac07bdd94a4f779a P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:38.544116 28529 raft_consensus.cc:3037] T 038a5d3d5368433cac07bdd94a4f779a P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:38.546687 28529 raft_consensus.cc:504] T 038a5d3d5368433cac07bdd94a4f779a P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:38.547415 28529 leader_election.cc:288] T 038a5d3d5368433cac07bdd94a4f779a P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Requested vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:14:38.548027 26689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "038a5d3d5368433cac07bdd94a4f779a" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:14:38.548213 26556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "038a5d3d5368433cac07bdd94a4f779a" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:14:38.548307 26689 raft_consensus.cc:3037] T 038a5d3d5368433cac07bdd94a4f779a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:38.548481 26556 raft_consensus.cc:3037] T 038a5d3d5368433cac07bdd94a4f779a P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:38.551366 26689 raft_consensus.cc:2454] T 038a5d3d5368433cac07bdd94a4f779a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:14:38.551366 26556 raft_consensus.cc:2454] T 038a5d3d5368433cac07bdd94a4f779a P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:14:38.551940 26363 leader_election.cc:302] T 038a5d3d5368433cac07bdd94a4f779a P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:14:38.552318 28529 raft_consensus.cc:2781] T 038a5d3d5368433cac07bdd94a4f779a P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:38.552517 28529 raft_consensus.cc:686] T 038a5d3d5368433cac07bdd94a4f779a P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Becoming Leader. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Running, Role: LEADER
I20240502 14:14:38.553210 28529 consensus_queue.cc:237] T 038a5d3d5368433cac07bdd94a4f779a P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:38.557014 26277 catalog_manager.cc:5484] T 038a5d3d5368433cac07bdd94a4f779a P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: term changed from 0 to 1, leader changed from <none> to 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193). New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:14:38.638687 28566 raft_consensus.cc:482] T f67582dfce354a9b92078ab0b5799194 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:38.638938 28566 raft_consensus.cc:504] T f67582dfce354a9b92078ab0b5799194 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:38.639904 28566 leader_election.cc:288] T f67582dfce354a9b92078ab0b5799194 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:14:38.640587 26556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f67582dfce354a9b92078ab0b5799194" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:38.640594 26689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f67582dfce354a9b92078ab0b5799194" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:14:38.640816 26556 raft_consensus.cc:2454] T f67582dfce354a9b92078ab0b5799194 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 0.
I20240502 14:14:38.640815 26689 raft_consensus.cc:2454] T f67582dfce354a9b92078ab0b5799194 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 0.
I20240502 14:14:38.641242 26754 leader_election.cc:302] T f67582dfce354a9b92078ab0b5799194 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:38.641534 28566 raft_consensus.cc:2781] T f67582dfce354a9b92078ab0b5799194 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:38.641661 28566 raft_consensus.cc:482] T f67582dfce354a9b92078ab0b5799194 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:38.641763 28566 raft_consensus.cc:3037] T f67582dfce354a9b92078ab0b5799194 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:38.644035 28566 raft_consensus.cc:504] T f67582dfce354a9b92078ab0b5799194 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:38.644667 28566 leader_election.cc:288] T f67582dfce354a9b92078ab0b5799194 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Requested vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:14:38.645248 26556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f67582dfce354a9b92078ab0b5799194" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:14:38.645263 26689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f67582dfce354a9b92078ab0b5799194" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:14:38.645478 26556 raft_consensus.cc:3037] T f67582dfce354a9b92078ab0b5799194 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:38.645476 26689 raft_consensus.cc:3037] T f67582dfce354a9b92078ab0b5799194 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:38.648378 26689 raft_consensus.cc:2454] T f67582dfce354a9b92078ab0b5799194 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
I20240502 14:14:38.648377 26556 raft_consensus.cc:2454] T f67582dfce354a9b92078ab0b5799194 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
I20240502 14:14:38.648931 26755 leader_election.cc:302] T f67582dfce354a9b92078ab0b5799194 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:38.649331 28566 raft_consensus.cc:2781] T f67582dfce354a9b92078ab0b5799194 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:38.649508 28566 raft_consensus.cc:686] T f67582dfce354a9b92078ab0b5799194 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Becoming Leader. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Running, Role: LEADER
I20240502 14:14:38.649806 28566 consensus_queue.cc:237] T f67582dfce354a9b92078ab0b5799194 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:38.653174 26277 catalog_manager.cc:5484] T f67582dfce354a9b92078ab0b5799194 P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196). New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:14:38.805759 28612 raft_consensus.cc:482] T 6f3334b24a3145e0b5498d8d97eabb1f P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:38.805944 28612 raft_consensus.cc:504] T 6f3334b24a3145e0b5498d8d97eabb1f P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:38.806852 28612 leader_election.cc:288] T 6f3334b24a3145e0b5498d8d97eabb1f P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:14:38.807384 26689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f3334b24a3145e0b5498d8d97eabb1f" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:14:38.807453 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f3334b24a3145e0b5498d8d97eabb1f" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:14:38.807616 26689 raft_consensus.cc:2454] T 6f3334b24a3145e0b5498d8d97eabb1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 0.
I20240502 14:14:38.807689 26426 raft_consensus.cc:2454] T 6f3334b24a3145e0b5498d8d97eabb1f P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 0.
I20240502 14:14:38.808071 26494 leader_election.cc:302] T 6f3334b24a3145e0b5498d8d97eabb1f P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:38.808351 28612 raft_consensus.cc:2781] T 6f3334b24a3145e0b5498d8d97eabb1f P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:38.808485 28612 raft_consensus.cc:482] T 6f3334b24a3145e0b5498d8d97eabb1f P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:38.808569 28612 raft_consensus.cc:3037] T 6f3334b24a3145e0b5498d8d97eabb1f P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:38.810386 28612 raft_consensus.cc:504] T 6f3334b24a3145e0b5498d8d97eabb1f P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:38.810926 28612 leader_election.cc:288] T 6f3334b24a3145e0b5498d8d97eabb1f P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Requested vote from peers 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:14:38.811385 26689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f3334b24a3145e0b5498d8d97eabb1f" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:14:38.811451 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f3334b24a3145e0b5498d8d97eabb1f" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:14:38.811565 26689 raft_consensus.cc:3037] T 6f3334b24a3145e0b5498d8d97eabb1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:38.811656 26426 raft_consensus.cc:3037] T 6f3334b24a3145e0b5498d8d97eabb1f P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:38.813778 26689 raft_consensus.cc:2454] T 6f3334b24a3145e0b5498d8d97eabb1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:14:38.813879 26426 raft_consensus.cc:2454] T 6f3334b24a3145e0b5498d8d97eabb1f P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:14:38.814337 26493 leader_election.cc:302] T 6f3334b24a3145e0b5498d8d97eabb1f P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:38.814630 28612 raft_consensus.cc:2781] T 6f3334b24a3145e0b5498d8d97eabb1f P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:38.814795 28612 raft_consensus.cc:686] T 6f3334b24a3145e0b5498d8d97eabb1f P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Becoming Leader. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Running, Role: LEADER
I20240502 14:14:38.815086 28612 consensus_queue.cc:237] T 6f3334b24a3145e0b5498d8d97eabb1f P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:38.818499 26277 catalog_manager.cc:5484] T 6f3334b24a3145e0b5498d8d97eabb1f P ba187e38a2754d60a12de9ac4856430a reported cstate change: term changed from 0 to 1, leader changed from <none> to ba187e38a2754d60a12de9ac4856430a (127.19.94.194). New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:14:39.054198 28590 consensus_queue.cc:1035] T 038a5d3d5368433cac07bdd94a4f779a P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:39.063221 28566 consensus_queue.cc:1035] T f67582dfce354a9b92078ab0b5799194 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:39.064759 28590 consensus_queue.cc:1035] T 038a5d3d5368433cac07bdd94a4f779a P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:39.071166 28623 consensus_queue.cc:1035] T f67582dfce354a9b92078ab0b5799194 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:39.266903 28615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:39.267279 28615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:39.275315 28615 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_14'
I20240502 14:14:39.355356 28612 consensus_queue.cc:1035] T 6f3334b24a3145e0b5498d8d97eabb1f P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:39.364264 28565 consensus_queue.cc:1035] T 6f3334b24a3145e0b5498d8d97eabb1f P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 255.871 ms
time per row: 0.127936 ms
I20240502 14:14:39.693614 26277 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalance_test_table_15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:39.704684 26798 tablet_service.cc:1467] Processing CreateTablet for tablet 6b4745b1153b4acd999ac3a5c137ff53 (DEFAULT_TABLE table=rebalance_test_table_15 [id=ff3107a4d4c94012a52f0cd0ca9ef235]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:39.704865 26406 tablet_service.cc:1467] Processing CreateTablet for tablet 6b4745b1153b4acd999ac3a5c137ff53 (DEFAULT_TABLE table=rebalance_test_table_15 [id=ff3107a4d4c94012a52f0cd0ca9ef235]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:39.705206 26798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b4745b1153b4acd999ac3a5c137ff53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:39.705334 26406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b4745b1153b4acd999ac3a5c137ff53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:39.709514 26799 tablet_service.cc:1467] Processing CreateTablet for tablet 533b42de2de14abc96c00a928d5deed1 (DEFAULT_TABLE table=rebalance_test_table_15 [id=ff3107a4d4c94012a52f0cd0ca9ef235]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:39.709790 26667 tablet_service.cc:1467] Processing CreateTablet for tablet 62218b2a597e4fb188ced6d2b9a83a59 (DEFAULT_TABLE table=rebalance_test_table_15 [id=ff3107a4d4c94012a52f0cd0ca9ef235]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:39.710011 26799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 533b42de2de14abc96c00a928d5deed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:39.710227 26667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62218b2a597e4fb188ced6d2b9a83a59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:39.712896 26797 tablet_service.cc:1467] Processing CreateTablet for tablet 62218b2a597e4fb188ced6d2b9a83a59 (DEFAULT_TABLE table=rebalance_test_table_15 [id=ff3107a4d4c94012a52f0cd0ca9ef235]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:39.713343 26797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62218b2a597e4fb188ced6d2b9a83a59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:39.716010 26669 tablet_service.cc:1467] Processing CreateTablet for tablet 533b42de2de14abc96c00a928d5deed1 (DEFAULT_TABLE table=rebalance_test_table_15 [id=ff3107a4d4c94012a52f0cd0ca9ef235]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:39.716471 26669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 533b42de2de14abc96c00a928d5deed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:39.720467 28682 tablet_bootstrap.cc:492] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:39.721514 28681 tablet_bootstrap.cc:492] T 6b4745b1153b4acd999ac3a5c137ff53 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:39.722302 28682 tablet_bootstrap.cc:654] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:39.723814 28681 tablet_bootstrap.cc:654] T 6b4745b1153b4acd999ac3a5c137ff53 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:39.725594 26406 tablet_service.cc:1467] Processing CreateTablet for tablet 533b42de2de14abc96c00a928d5deed1 (DEFAULT_TABLE table=rebalance_test_table_15 [id=ff3107a4d4c94012a52f0cd0ca9ef235]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:39.726028 26406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 533b42de2de14abc96c00a928d5deed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:39.726229 26538 tablet_service.cc:1467] Processing CreateTablet for tablet 62218b2a597e4fb188ced6d2b9a83a59 (DEFAULT_TABLE table=rebalance_test_table_15 [id=ff3107a4d4c94012a52f0cd0ca9ef235]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:39.726820 26538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62218b2a597e4fb188ced6d2b9a83a59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:39.728768 26536 tablet_service.cc:1467] Processing CreateTablet for tablet 6b4745b1153b4acd999ac3a5c137ff53 (DEFAULT_TABLE table=rebalance_test_table_15 [id=ff3107a4d4c94012a52f0cd0ca9ef235]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:39.729142 26536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b4745b1153b4acd999ac3a5c137ff53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:39.743768 28682 tablet_bootstrap.cc:492] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:39.743942 28682 ts_tablet_manager.cc:1367] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.000s
I20240502 14:14:39.744800 28682 raft_consensus.cc:348] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:39.745208 28682 raft_consensus.cc:374] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:39.745369 28682 raft_consensus.cc:729] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:39.745658 28682 consensus_queue.cc:260] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:39.746372 28682 ts_tablet_manager.cc:1398] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:39.746628 28685 tablet_bootstrap.cc:492] T 6b4745b1153b4acd999ac3a5c137ff53 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:39.748986 28685 tablet_bootstrap.cc:654] T 6b4745b1153b4acd999ac3a5c137ff53 P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:39.758469 28685 tablet_bootstrap.cc:492] T 6b4745b1153b4acd999ac3a5c137ff53 P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:39.758837 28685 ts_tablet_manager.cc:1367] T 6b4745b1153b4acd999ac3a5c137ff53 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20240502 14:14:39.759750 28682 tablet_bootstrap.cc:492] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:39.761323 28685 raft_consensus.cc:348] T 6b4745b1153b4acd999ac3a5c137ff53 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:39.761621 28685 raft_consensus.cc:374] T 6b4745b1153b4acd999ac3a5c137ff53 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:39.761773 28685 raft_consensus.cc:729] T 6b4745b1153b4acd999ac3a5c137ff53 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:39.761853 28682 tablet_bootstrap.cc:654] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:39.762301 28685 consensus_queue.cc:260] T 6b4745b1153b4acd999ac3a5c137ff53 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:39.763001 28685 ts_tablet_manager.cc:1398] T 6b4745b1153b4acd999ac3a5c137ff53 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:14:39.765017 28687 tablet_bootstrap.cc:492] T 533b42de2de14abc96c00a928d5deed1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:39.767629 28612 raft_consensus.cc:482] T 6b4745b1153b4acd999ac3a5c137ff53 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:39.767798 28612 raft_consensus.cc:504] T 6b4745b1153b4acd999ac3a5c137ff53 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:39.768610 28612 leader_election.cc:288] T 6b4745b1153b4acd999ac3a5c137ff53 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:14:39.768873 28685 tablet_bootstrap.cc:492] T 62218b2a597e4fb188ced6d2b9a83a59 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:39.769634 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4745b1153b4acd999ac3a5c137ff53" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
W20240502 14:14:39.770630 26493 leader_election.cc:341] T 6b4745b1153b4acd999ac3a5c137ff53 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:39.770723 28685 tablet_bootstrap.cc:654] T 62218b2a597e4fb188ced6d2b9a83a59 P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:39.771301 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4745b1153b4acd999ac3a5c137ff53" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
W20240502 14:14:39.771960 26494 leader_election.cc:341] T 6b4745b1153b4acd999ac3a5c137ff53 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:39.772090 26494 leader_election.cc:302] T 6b4745b1153b4acd999ac3a5c137ff53 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27
I20240502 14:14:39.772511 28612 raft_consensus.cc:2726] T 6b4745b1153b4acd999ac3a5c137ff53 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:14:39.773187 28687 tablet_bootstrap.cc:654] T 533b42de2de14abc96c00a928d5deed1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:39.774870 28681 tablet_bootstrap.cc:492] T 6b4745b1153b4acd999ac3a5c137ff53 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:39.775027 28681 ts_tablet_manager.cc:1367] T 6b4745b1153b4acd999ac3a5c137ff53 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.054s user 0.023s sys 0.015s
I20240502 14:14:39.775790 28681 raft_consensus.cc:348] T 6b4745b1153b4acd999ac3a5c137ff53 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:39.775996 28681 raft_consensus.cc:374] T 6b4745b1153b4acd999ac3a5c137ff53 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:39.776084 28681 raft_consensus.cc:729] T 6b4745b1153b4acd999ac3a5c137ff53 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:39.776325 28681 consensus_queue.cc:260] T 6b4745b1153b4acd999ac3a5c137ff53 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:39.776928 28681 ts_tablet_manager.cc:1398] T 6b4745b1153b4acd999ac3a5c137ff53 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:39.777238 28681 tablet_bootstrap.cc:492] T 533b42de2de14abc96c00a928d5deed1 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:39.778950 28681 tablet_bootstrap.cc:654] T 533b42de2de14abc96c00a928d5deed1 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:39.782354 28682 tablet_bootstrap.cc:492] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:39.782603 28682 ts_tablet_manager.cc:1367] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.003s
I20240502 14:14:39.784106 28682 raft_consensus.cc:348] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:39.784453 28682 raft_consensus.cc:374] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:39.784593 28682 raft_consensus.cc:729] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:39.784916 28682 consensus_queue.cc:260] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:39.790160 28682 ts_tablet_manager.cc:1398] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240502 14:14:39.790669 28685 tablet_bootstrap.cc:492] T 62218b2a597e4fb188ced6d2b9a83a59 P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:39.790932 28685 ts_tablet_manager.cc:1367] T 62218b2a597e4fb188ced6d2b9a83a59 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20240502 14:14:39.791955 28685 raft_consensus.cc:348] T 62218b2a597e4fb188ced6d2b9a83a59 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:39.792296 28685 raft_consensus.cc:374] T 62218b2a597e4fb188ced6d2b9a83a59 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:39.792469 28685 raft_consensus.cc:729] T 62218b2a597e4fb188ced6d2b9a83a59 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:39.792809 28685 consensus_queue.cc:260] T 62218b2a597e4fb188ced6d2b9a83a59 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:39.793696 28685 ts_tablet_manager.cc:1398] T 62218b2a597e4fb188ced6d2b9a83a59 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:39.795570 28687 tablet_bootstrap.cc:492] T 533b42de2de14abc96c00a928d5deed1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:39.796838 28682 tablet_bootstrap.cc:492] T 62218b2a597e4fb188ced6d2b9a83a59 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:39.800474 28687 ts_tablet_manager.cc:1367] T 533b42de2de14abc96c00a928d5deed1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.002s
I20240502 14:14:39.802762 28687 raft_consensus.cc:348] T 533b42de2de14abc96c00a928d5deed1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:39.803217 28687 raft_consensus.cc:374] T 533b42de2de14abc96c00a928d5deed1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:39.803427 28687 raft_consensus.cc:729] T 533b42de2de14abc96c00a928d5deed1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:39.803776 28687 consensus_queue.cc:260] T 533b42de2de14abc96c00a928d5deed1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:39.804652 28687 ts_tablet_manager.cc:1398] T 533b42de2de14abc96c00a928d5deed1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 14:14:39.806001 28682 tablet_bootstrap.cc:654] T 62218b2a597e4fb188ced6d2b9a83a59 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:39.807142 28687 tablet_bootstrap.cc:492] T 62218b2a597e4fb188ced6d2b9a83a59 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:39.808174 28681 tablet_bootstrap.cc:492] T 533b42de2de14abc96c00a928d5deed1 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:39.808434 28681 ts_tablet_manager.cc:1367] T 533b42de2de14abc96c00a928d5deed1 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.031s user 0.018s sys 0.011s
I20240502 14:14:39.809471 28681 raft_consensus.cc:348] T 533b42de2de14abc96c00a928d5deed1 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:39.809728 28681 raft_consensus.cc:374] T 533b42de2de14abc96c00a928d5deed1 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:39.809803 28681 raft_consensus.cc:729] T 533b42de2de14abc96c00a928d5deed1 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:39.810207 28681 consensus_queue.cc:260] T 533b42de2de14abc96c00a928d5deed1 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:39.810614 28687 tablet_bootstrap.cc:654] T 62218b2a597e4fb188ced6d2b9a83a59 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:39.812822 28682 tablet_bootstrap.cc:492] T 62218b2a597e4fb188ced6d2b9a83a59 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:39.813035 28682 ts_tablet_manager.cc:1367] T 62218b2a597e4fb188ced6d2b9a83a59 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.002s
I20240502 14:14:39.813946 28566 raft_consensus.cc:482] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:39.814097 28566 raft_consensus.cc:504] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:39.814222 28682 raft_consensus.cc:348] T 62218b2a597e4fb188ced6d2b9a83a59 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:39.814376 28682 raft_consensus.cc:374] T 62218b2a597e4fb188ced6d2b9a83a59 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:39.814465 28682 raft_consensus.cc:729] T 62218b2a597e4fb188ced6d2b9a83a59 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:39.814683 28682 consensus_queue.cc:260] T 62218b2a597e4fb188ced6d2b9a83a59 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:39.814970 28566 leader_election.cc:288] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:14:39.815284 28682 ts_tablet_manager.cc:1398] T 62218b2a597e4fb188ced6d2b9a83a59 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:39.815774 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533b42de2de14abc96c00a928d5deed1" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:14:39.816067 26426 raft_consensus.cc:2454] T 533b42de2de14abc96c00a928d5deed1 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 0.
I20240502 14:14:39.816148 26689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533b42de2de14abc96c00a928d5deed1" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:14:39.816382 26689 raft_consensus.cc:2454] T 533b42de2de14abc96c00a928d5deed1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 0.
I20240502 14:14:39.816692 26754 leader_election.cc:302] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:39.816777 28681 ts_tablet_manager.cc:1398] T 533b42de2de14abc96c00a928d5deed1 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240502 14:14:39.817019 28623 raft_consensus.cc:2781] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:39.817140 28623 raft_consensus.cc:482] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:39.817242 28623 raft_consensus.cc:3037] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:39.818087 28687 tablet_bootstrap.cc:492] T 62218b2a597e4fb188ced6d2b9a83a59 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:39.818470 28687 ts_tablet_manager.cc:1367] T 62218b2a597e4fb188ced6d2b9a83a59 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.004s
I20240502 14:14:39.819456 28687 raft_consensus.cc:348] T 62218b2a597e4fb188ced6d2b9a83a59 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:39.819825 28687 raft_consensus.cc:374] T 62218b2a597e4fb188ced6d2b9a83a59 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:39.819955 28687 raft_consensus.cc:729] T 62218b2a597e4fb188ced6d2b9a83a59 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:39.820277 28687 consensus_queue.cc:260] T 62218b2a597e4fb188ced6d2b9a83a59 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:39.820461 28623 raft_consensus.cc:504] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:39.821727 28687 ts_tablet_manager.cc:1398] T 62218b2a597e4fb188ced6d2b9a83a59 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:39.821731 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533b42de2de14abc96c00a928d5deed1" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:14:39.821976 26426 raft_consensus.cc:3037] T 533b42de2de14abc96c00a928d5deed1 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:39.822615 26689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533b42de2de14abc96c00a928d5deed1" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:14:39.822818 26689 raft_consensus.cc:3037] T 533b42de2de14abc96c00a928d5deed1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:39.823336 28623 leader_election.cc:288] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Requested vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:14:39.824990 26426 raft_consensus.cc:2454] T 533b42de2de14abc96c00a928d5deed1 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
I20240502 14:14:39.825507 26754 leader_election.cc:302] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:39.825682 26689 raft_consensus.cc:2454] T 533b42de2de14abc96c00a928d5deed1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
I20240502 14:14:39.825889 28623 raft_consensus.cc:2781] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:39.826177 28623 raft_consensus.cc:686] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Becoming Leader. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Running, Role: LEADER
I20240502 14:14:39.826483 28623 consensus_queue.cc:237] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:39.829525 26278 catalog_manager.cc:5484] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196). New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:14:39.881577 28612 raft_consensus.cc:482] T 62218b2a597e4fb188ced6d2b9a83a59 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:39.881747 28612 raft_consensus.cc:504] T 62218b2a597e4fb188ced6d2b9a83a59 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:39.882308 28612 leader_election.cc:288] T 62218b2a597e4fb188ced6d2b9a83a59 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:14:39.882763 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62218b2a597e4fb188ced6d2b9a83a59" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:14:39.882891 26689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62218b2a597e4fb188ced6d2b9a83a59" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:14:39.883005 26818 raft_consensus.cc:2454] T 62218b2a597e4fb188ced6d2b9a83a59 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 0.
I20240502 14:14:39.883164 26689 raft_consensus.cc:2454] T 62218b2a597e4fb188ced6d2b9a83a59 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 0.
I20240502 14:14:39.883438 26494 leader_election.cc:302] T 62218b2a597e4fb188ced6d2b9a83a59 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:39.883738 28612 raft_consensus.cc:2781] T 62218b2a597e4fb188ced6d2b9a83a59 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:39.883855 28612 raft_consensus.cc:482] T 62218b2a597e4fb188ced6d2b9a83a59 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:39.883958 28612 raft_consensus.cc:3037] T 62218b2a597e4fb188ced6d2b9a83a59 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:39.886505 28612 raft_consensus.cc:504] T 62218b2a597e4fb188ced6d2b9a83a59 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:39.887084 28612 leader_election.cc:288] T 62218b2a597e4fb188ced6d2b9a83a59 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Requested vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:14:39.887643 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62218b2a597e4fb188ced6d2b9a83a59" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:14:39.887785 26689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62218b2a597e4fb188ced6d2b9a83a59" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:14:39.887876 26818 raft_consensus.cc:3037] T 62218b2a597e4fb188ced6d2b9a83a59 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:39.888023 26689 raft_consensus.cc:3037] T 62218b2a597e4fb188ced6d2b9a83a59 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:39.890857 26689 raft_consensus.cc:2454] T 62218b2a597e4fb188ced6d2b9a83a59 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:14:39.890875 26818 raft_consensus.cc:2454] T 62218b2a597e4fb188ced6d2b9a83a59 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:14:39.891422 26494 leader_election.cc:302] T 62218b2a597e4fb188ced6d2b9a83a59 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:39.891798 28612 raft_consensus.cc:2781] T 62218b2a597e4fb188ced6d2b9a83a59 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:39.891988 28612 raft_consensus.cc:686] T 62218b2a597e4fb188ced6d2b9a83a59 P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Becoming Leader. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Running, Role: LEADER
I20240502 14:14:39.892295 28612 consensus_queue.cc:237] T 62218b2a597e4fb188ced6d2b9a83a59 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:39.895905 26278 catalog_manager.cc:5484] T 62218b2a597e4fb188ced6d2b9a83a59 P ba187e38a2754d60a12de9ac4856430a reported cstate change: term changed from 0 to 1, leader changed from <none> to ba187e38a2754d60a12de9ac4856430a (127.19.94.194). New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:14:40.007670 28623 raft_consensus.cc:482] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:40.007899 28623 raft_consensus.cc:504] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:40.008572 28623 leader_election.cc:288] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:14:40.009140 26556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4745b1153b4acd999ac3a5c137ff53" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:40.009140 26424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4745b1153b4acd999ac3a5c137ff53" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:14:40.009404 26556 raft_consensus.cc:2454] T 6b4745b1153b4acd999ac3a5c137ff53 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 0.
I20240502 14:14:40.009423 26424 raft_consensus.cc:2454] T 6b4745b1153b4acd999ac3a5c137ff53 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 0.
I20240502 14:14:40.010366 26754 leader_election.cc:302] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:40.010879 28623 raft_consensus.cc:2781] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:40.010998 28623 raft_consensus.cc:482] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:40.011126 28623 raft_consensus.cc:3037] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:40.013684 28623 raft_consensus.cc:504] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:40.014328 28623 leader_election.cc:288] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Requested vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:14:40.014755 26424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4745b1153b4acd999ac3a5c137ff53" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:14:40.014815 26556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4745b1153b4acd999ac3a5c137ff53" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:14:40.014940 26424 raft_consensus.cc:3037] T 6b4745b1153b4acd999ac3a5c137ff53 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:40.015023 26556 raft_consensus.cc:3037] T 6b4745b1153b4acd999ac3a5c137ff53 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:40.017123 26424 raft_consensus.cc:2454] T 6b4745b1153b4acd999ac3a5c137ff53 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
I20240502 14:14:40.017220 26556 raft_consensus.cc:2454] T 6b4745b1153b4acd999ac3a5c137ff53 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
I20240502 14:14:40.017607 26754 leader_election.cc:302] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:40.017926 28623 raft_consensus.cc:2781] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:40.018121 28623 raft_consensus.cc:686] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Becoming Leader. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Running, Role: LEADER
I20240502 14:14:40.018517 28623 consensus_queue.cc:237] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:40.033720 26278 catalog_manager.cc:5484] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196). New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:14:40.271711 28566 consensus_queue.cc:1035] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:40.276536 28566 consensus_queue.cc:1035] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:40.405917 28612 consensus_queue.cc:1035] T 62218b2a597e4fb188ced6d2b9a83a59 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:40.411425 28612 consensus_queue.cc:1035] T 62218b2a597e4fb188ced6d2b9a83a59 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:40.444697 28697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:40.445065 28697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:40.453742 28697 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_15'
I20240502 14:14:40.534662 28566 consensus_queue.cc:1035] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:40.543833 28623 consensus_queue.cc:1035] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 310.443 ms
time per row: 0.155221 ms
I20240502 14:14:40.935920 26276 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalance_test_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:40.945960 26406 tablet_service.cc:1467] Processing CreateTablet for tablet 82737d75c8a64efe98cab1e21160d9b6 (DEFAULT_TABLE table=rebalance_test_table_16 [id=f1e711a462c54504b4fe5c866615f824]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:40.946003 26797 tablet_service.cc:1467] Processing CreateTablet for tablet 82737d75c8a64efe98cab1e21160d9b6 (DEFAULT_TABLE table=rebalance_test_table_16 [id=f1e711a462c54504b4fe5c866615f824]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:40.946449 26797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82737d75c8a64efe98cab1e21160d9b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:40.946467 26406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82737d75c8a64efe98cab1e21160d9b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:40.946604 26538 tablet_service.cc:1467] Processing CreateTablet for tablet 82737d75c8a64efe98cab1e21160d9b6 (DEFAULT_TABLE table=rebalance_test_table_16 [id=f1e711a462c54504b4fe5c866615f824]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:40.947010 26538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82737d75c8a64efe98cab1e21160d9b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:40.948161 26669 tablet_service.cc:1467] Processing CreateTablet for tablet 2b50fc58a5384c6b8d2c78b88e0b3d95 (DEFAULT_TABLE table=rebalance_test_table_16 [id=f1e711a462c54504b4fe5c866615f824]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:40.948604 26669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b50fc58a5384c6b8d2c78b88e0b3d95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:40.953518 26405 tablet_service.cc:1467] Processing CreateTablet for tablet 2b50fc58a5384c6b8d2c78b88e0b3d95 (DEFAULT_TABLE table=rebalance_test_table_16 [id=f1e711a462c54504b4fe5c866615f824]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:40.954030 26405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b50fc58a5384c6b8d2c78b88e0b3d95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:40.954180 26536 tablet_service.cc:1467] Processing CreateTablet for tablet 2b50fc58a5384c6b8d2c78b88e0b3d95 (DEFAULT_TABLE table=rebalance_test_table_16 [id=f1e711a462c54504b4fe5c866615f824]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:40.954833 26536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b50fc58a5384c6b8d2c78b88e0b3d95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:40.957468 26798 tablet_service.cc:1467] Processing CreateTablet for tablet 5d8ed238257d46b9b78f0f8a19f2adba (DEFAULT_TABLE table=rebalance_test_table_16 [id=f1e711a462c54504b4fe5c866615f824]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:40.957754 26667 tablet_service.cc:1467] Processing CreateTablet for tablet 5d8ed238257d46b9b78f0f8a19f2adba (DEFAULT_TABLE table=rebalance_test_table_16 [id=f1e711a462c54504b4fe5c866615f824]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:40.957928 26798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d8ed238257d46b9b78f0f8a19f2adba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:40.958187 26667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d8ed238257d46b9b78f0f8a19f2adba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:40.962389 26537 tablet_service.cc:1467] Processing CreateTablet for tablet 5d8ed238257d46b9b78f0f8a19f2adba (DEFAULT_TABLE table=rebalance_test_table_16 [id=f1e711a462c54504b4fe5c866615f824]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:40.962872 26537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d8ed238257d46b9b78f0f8a19f2adba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:40.970806 28762 tablet_bootstrap.cc:492] T 82737d75c8a64efe98cab1e21160d9b6 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:40.972647 28762 tablet_bootstrap.cc:654] T 82737d75c8a64efe98cab1e21160d9b6 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:40.975337 28761 tablet_bootstrap.cc:492] T 82737d75c8a64efe98cab1e21160d9b6 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:40.975353 28763 tablet_bootstrap.cc:492] T 5d8ed238257d46b9b78f0f8a19f2adba P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:40.977207 28763 tablet_bootstrap.cc:654] T 5d8ed238257d46b9b78f0f8a19f2adba P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:40.979113 28760 tablet_bootstrap.cc:492] T 82737d75c8a64efe98cab1e21160d9b6 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:40.979200 28761 tablet_bootstrap.cc:654] T 82737d75c8a64efe98cab1e21160d9b6 P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:40.981201 28760 tablet_bootstrap.cc:654] T 82737d75c8a64efe98cab1e21160d9b6 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:40.981427 28762 tablet_bootstrap.cc:492] T 82737d75c8a64efe98cab1e21160d9b6 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:40.981599 28762 ts_tablet_manager.cc:1367] T 82737d75c8a64efe98cab1e21160d9b6 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.003s
I20240502 14:14:40.982100 28763 tablet_bootstrap.cc:492] T 5d8ed238257d46b9b78f0f8a19f2adba P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:40.982259 28763 ts_tablet_manager.cc:1367] T 5d8ed238257d46b9b78f0f8a19f2adba P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240502 14:14:40.983130 28763 raft_consensus.cc:348] T 5d8ed238257d46b9b78f0f8a19f2adba P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:40.983202 28762 raft_consensus.cc:348] T 82737d75c8a64efe98cab1e21160d9b6 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:40.983407 28761 tablet_bootstrap.cc:492] T 82737d75c8a64efe98cab1e21160d9b6 P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:40.983453 28763 raft_consensus.cc:374] T 5d8ed238257d46b9b78f0f8a19f2adba P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:40.983569 28761 ts_tablet_manager.cc:1367] T 82737d75c8a64efe98cab1e21160d9b6 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20240502 14:14:40.983577 28763 raft_consensus.cc:729] T 5d8ed238257d46b9b78f0f8a19f2adba P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:40.983620 28762 raft_consensus.cc:374] T 82737d75c8a64efe98cab1e21160d9b6 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:40.983798 28762 raft_consensus.cc:729] T 82737d75c8a64efe98cab1e21160d9b6 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:40.983837 28763 consensus_queue.cc:260] T 5d8ed238257d46b9b78f0f8a19f2adba P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:40.984418 28761 raft_consensus.cc:348] T 82737d75c8a64efe98cab1e21160d9b6 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:40.984627 28761 raft_consensus.cc:374] T 82737d75c8a64efe98cab1e21160d9b6 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:40.984639 28763 ts_tablet_manager.cc:1398] T 5d8ed238257d46b9b78f0f8a19f2adba P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:40.984720 28761 raft_consensus.cc:729] T 82737d75c8a64efe98cab1e21160d9b6 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:40.984964 28761 consensus_queue.cc:260] T 82737d75c8a64efe98cab1e21160d9b6 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:40.985214 28763 tablet_bootstrap.cc:492] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:40.985554 28761 ts_tablet_manager.cc:1398] T 82737d75c8a64efe98cab1e21160d9b6 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:14:40.985849 28761 tablet_bootstrap.cc:492] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:40.984100 28762 consensus_queue.cc:260] T 82737d75c8a64efe98cab1e21160d9b6 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:40.986780 28762 ts_tablet_manager.cc:1398] T 82737d75c8a64efe98cab1e21160d9b6 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20240502 14:14:40.987462 28762 tablet_bootstrap.cc:492] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:40.987597 28761 tablet_bootstrap.cc:654] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:40.990628 28762 tablet_bootstrap.cc:654] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:40.992431 28763 tablet_bootstrap.cc:654] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:40.998575 28760 tablet_bootstrap.cc:492] T 82737d75c8a64efe98cab1e21160d9b6 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:40.998694 28763 tablet_bootstrap.cc:492] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:40.998817 28760 ts_tablet_manager.cc:1367] T 82737d75c8a64efe98cab1e21160d9b6 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.004s
I20240502 14:14:40.998903 28763 ts_tablet_manager.cc:1367] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.007s
I20240502 14:14:40.999541 28763 raft_consensus.cc:348] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:40.999750 28763 raft_consensus.cc:374] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:40.999837 28763 raft_consensus.cc:729] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:40.999964 28760 raft_consensus.cc:348] T 82737d75c8a64efe98cab1e21160d9b6 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:41.000144 28763 consensus_queue.cc:260] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:41.000382 28760 raft_consensus.cc:374] T 82737d75c8a64efe98cab1e21160d9b6 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:41.000562 28760 raft_consensus.cc:729] T 82737d75c8a64efe98cab1e21160d9b6 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:41.000828 28763 ts_tablet_manager.cc:1398] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:41.000818 28760 consensus_queue.cc:260] T 82737d75c8a64efe98cab1e21160d9b6 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:41.001446 28760 ts_tablet_manager.cc:1398] T 82737d75c8a64efe98cab1e21160d9b6 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 14:14:41.007848 28760 tablet_bootstrap.cc:492] T 5d8ed238257d46b9b78f0f8a19f2adba P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:41.009502 28760 tablet_bootstrap.cc:654] T 5d8ed238257d46b9b78f0f8a19f2adba P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:41.010435 28628 raft_consensus.cc:482] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:41.010605 28628 raft_consensus.cc:504] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:41.011210 28761 tablet_bootstrap.cc:492] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:41.011400 28761 ts_tablet_manager.cc:1367] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.000s
I20240502 14:14:41.011662 28628 leader_election.cc:288] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:14:41.012260 28761 raft_consensus.cc:348] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:41.012543 28761 raft_consensus.cc:374] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:41.012638 28761 raft_consensus.cc:729] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:41.012838 26556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b50fc58a5384c6b8d2c78b88e0b3d95" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:41.012975 28761 consensus_queue.cc:260] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:41.013279 26556 raft_consensus.cc:2454] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:14:41.013973 26624 leader_election.cc:302] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:41.014281 28628 raft_consensus.cc:2781] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:41.014420 28628 raft_consensus.cc:482] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:41.014518 28628 raft_consensus.cc:3037] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:41.014467 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b50fc58a5384c6b8d2c78b88e0b3d95" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
W20240502 14:14:41.015156 26624 leader_election.cc:341] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:41.015499 28762 tablet_bootstrap.cc:492] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:41.015704 28762 ts_tablet_manager.cc:1367] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.028s user 0.015s sys 0.004s
I20240502 14:14:41.016983 28762 raft_consensus.cc:348] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:41.017273 28762 raft_consensus.cc:374] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:41.017356 28762 raft_consensus.cc:729] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:41.017642 28628 raft_consensus.cc:504] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:41.017668 28762 consensus_queue.cc:260] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:41.018507 28762 ts_tablet_manager.cc:1398] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:41.018784 28628 leader_election.cc:288] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Requested vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:14:41.021456 28760 tablet_bootstrap.cc:492] T 5d8ed238257d46b9b78f0f8a19f2adba P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:41.021623 26556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b50fc58a5384c6b8d2c78b88e0b3d95" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:14:41.021878 26556 raft_consensus.cc:3037] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:41.023272 28760 ts_tablet_manager.cc:1367] T 5d8ed238257d46b9b78f0f8a19f2adba P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.004s
I20240502 14:14:41.024287 28760 raft_consensus.cc:348] T 5d8ed238257d46b9b78f0f8a19f2adba P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:41.024610 28760 raft_consensus.cc:374] T 5d8ed238257d46b9b78f0f8a19f2adba P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:41.024694 28760 raft_consensus.cc:729] T 5d8ed238257d46b9b78f0f8a19f2adba P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:41.024974 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b50fc58a5384c6b8d2c78b88e0b3d95" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:14:41.024986 28760 consensus_queue.cc:260] T 5d8ed238257d46b9b78f0f8a19f2adba P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:41.025215 26426 raft_consensus.cc:3037] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:41.026686 28760 ts_tablet_manager.cc:1398] T 5d8ed238257d46b9b78f0f8a19f2adba P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:41.027689 26426 raft_consensus.cc:2454] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:14:41.028127 26624 leader_election.cc:302] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:14:41.028519 28628 raft_consensus.cc:2781] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:41.028651 28628 raft_consensus.cc:686] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Becoming Leader. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Running, Role: LEADER
I20240502 14:14:41.028924 28628 consensus_queue.cc:237] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:41.030015 28761 ts_tablet_manager.cc:1398] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.019s user 0.003s sys 0.000s
I20240502 14:14:41.030180 28748 raft_consensus.cc:482] T 82737d75c8a64efe98cab1e21160d9b6 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:41.030357 28748 raft_consensus.cc:504] T 82737d75c8a64efe98cab1e21160d9b6 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:41.031019 28761 tablet_bootstrap.cc:492] T 5d8ed238257d46b9b78f0f8a19f2adba P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:41.031081 28748 leader_election.cc:288] T 82737d75c8a64efe98cab1e21160d9b6 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:14:41.031816 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82737d75c8a64efe98cab1e21160d9b6" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:41.032092 26558 raft_consensus.cc:2454] T 82737d75c8a64efe98cab1e21160d9b6 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 0.
I20240502 14:14:41.033088 26754 leader_election.cc:302] T 82737d75c8a64efe98cab1e21160d9b6 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:41.033421 28748 raft_consensus.cc:2781] T 82737d75c8a64efe98cab1e21160d9b6 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:41.033547 28748 raft_consensus.cc:482] T 82737d75c8a64efe98cab1e21160d9b6 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:41.033650 28748 raft_consensus.cc:3037] T 82737d75c8a64efe98cab1e21160d9b6 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:41.033730 28761 tablet_bootstrap.cc:654] T 5d8ed238257d46b9b78f0f8a19f2adba P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:41.033603 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82737d75c8a64efe98cab1e21160d9b6" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:14:41.033958 26426 raft_consensus.cc:2454] T 82737d75c8a64efe98cab1e21160d9b6 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 0.
I20240502 14:14:41.036489 28748 raft_consensus.cc:504] T 82737d75c8a64efe98cab1e21160d9b6 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:41.037074 28748 leader_election.cc:288] T 82737d75c8a64efe98cab1e21160d9b6 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Requested vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:14:41.038169 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82737d75c8a64efe98cab1e21160d9b6" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:14:41.038515 26426 raft_consensus.cc:3037] T 82737d75c8a64efe98cab1e21160d9b6 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:41.038554 26276 catalog_manager.cc:5484] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195). New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:14:41.041538 26426 raft_consensus.cc:2454] T 82737d75c8a64efe98cab1e21160d9b6 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
I20240502 14:14:41.042075 26754 leader_election.cc:302] T 82737d75c8a64efe98cab1e21160d9b6 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:41.031827 26556 raft_consensus.cc:2454] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:14:41.037886 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82737d75c8a64efe98cab1e21160d9b6" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:14:41.042564 26558 raft_consensus.cc:3037] T 82737d75c8a64efe98cab1e21160d9b6 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:41.044040 28761 tablet_bootstrap.cc:492] T 5d8ed238257d46b9b78f0f8a19f2adba P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:41.044191 28761 ts_tablet_manager.cc:1367] T 5d8ed238257d46b9b78f0f8a19f2adba P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240502 14:14:41.044839 28748 raft_consensus.cc:2781] T 82737d75c8a64efe98cab1e21160d9b6 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:41.045087 28748 raft_consensus.cc:686] T 82737d75c8a64efe98cab1e21160d9b6 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Becoming Leader. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Running, Role: LEADER
I20240502 14:14:41.045466 28748 consensus_queue.cc:237] T 82737d75c8a64efe98cab1e21160d9b6 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:41.045835 26558 raft_consensus.cc:2454] T 82737d75c8a64efe98cab1e21160d9b6 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
I20240502 14:14:41.045843 28761 raft_consensus.cc:348] T 5d8ed238257d46b9b78f0f8a19f2adba P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:41.046130 28761 raft_consensus.cc:374] T 5d8ed238257d46b9b78f0f8a19f2adba P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:41.046245 28761 raft_consensus.cc:729] T 5d8ed238257d46b9b78f0f8a19f2adba P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:41.046474 28761 consensus_queue.cc:260] T 5d8ed238257d46b9b78f0f8a19f2adba P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:41.047076 28761 ts_tablet_manager.cc:1398] T 5d8ed238257d46b9b78f0f8a19f2adba P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:14:41.051808 26282 catalog_manager.cc:5484] T 82737d75c8a64efe98cab1e21160d9b6 P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196). New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:14:41.106567 28628 raft_consensus.cc:482] T 5d8ed238257d46b9b78f0f8a19f2adba P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:41.106832 28628 raft_consensus.cc:504] T 5d8ed238257d46b9b78f0f8a19f2adba P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:41.107590 28628 leader_election.cc:288] T 5d8ed238257d46b9b78f0f8a19f2adba P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:14:41.108366 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d8ed238257d46b9b78f0f8a19f2adba" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:14:41.108362 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d8ed238257d46b9b78f0f8a19f2adba" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:41.108655 26558 raft_consensus.cc:2454] T 5d8ed238257d46b9b78f0f8a19f2adba P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:14:41.108683 26818 raft_consensus.cc:2454] T 5d8ed238257d46b9b78f0f8a19f2adba P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:14:41.109207 26624 leader_election.cc:302] T 5d8ed238257d46b9b78f0f8a19f2adba P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:41.109544 28586 raft_consensus.cc:2781] T 5d8ed238257d46b9b78f0f8a19f2adba P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:41.109653 28586 raft_consensus.cc:482] T 5d8ed238257d46b9b78f0f8a19f2adba P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:41.109784 28586 raft_consensus.cc:3037] T 5d8ed238257d46b9b78f0f8a19f2adba P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:41.112303 28586 raft_consensus.cc:504] T 5d8ed238257d46b9b78f0f8a19f2adba P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:41.113155 28586 leader_election.cc:288] T 5d8ed238257d46b9b78f0f8a19f2adba P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Requested vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:14:41.113766 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d8ed238257d46b9b78f0f8a19f2adba" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:14:41.114027 26818 raft_consensus.cc:3037] T 5d8ed238257d46b9b78f0f8a19f2adba P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:41.114091 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d8ed238257d46b9b78f0f8a19f2adba" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:14:41.114308 26558 raft_consensus.cc:3037] T 5d8ed238257d46b9b78f0f8a19f2adba P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:41.116180 26818 raft_consensus.cc:2454] T 5d8ed238257d46b9b78f0f8a19f2adba P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:14:41.116199 26558 raft_consensus.cc:2454] T 5d8ed238257d46b9b78f0f8a19f2adba P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:14:41.116739 26624 leader_election.cc:302] T 5d8ed238257d46b9b78f0f8a19f2adba P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, ba187e38a2754d60a12de9ac4856430a, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:41.117393 28586 raft_consensus.cc:2781] T 5d8ed238257d46b9b78f0f8a19f2adba P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:41.117592 28586 raft_consensus.cc:686] T 5d8ed238257d46b9b78f0f8a19f2adba P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Becoming Leader. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Running, Role: LEADER
I20240502 14:14:41.117866 28586 consensus_queue.cc:237] T 5d8ed238257d46b9b78f0f8a19f2adba P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:41.121672 26282 catalog_manager.cc:5484] T 5d8ed238257d46b9b78f0f8a19f2adba P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195). New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:14:41.464701 28586 consensus_queue.cc:1035] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:41.471292 28628 consensus_queue.cc:1035] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:41.557047 28776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:41.557430 28776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 14:14:41.558729 28748 consensus_queue.cc:1035] T 82737d75c8a64efe98cab1e21160d9b6 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:41.567512 28748 consensus_queue.cc:1035] T 82737d75c8a64efe98cab1e21160d9b6 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:41.570952 28776 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_16'
I20240502 14:14:41.627733 28628 consensus_queue.cc:1035] T 5d8ed238257d46b9b78f0f8a19f2adba P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:41.637379 28809 consensus_queue.cc:1035] T 5d8ed238257d46b9b78f0f8a19f2adba P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 359.91 ms
time per row: 0.179955 ms
I20240502 14:14:42.108614 26280 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalance_test_table_17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:42.121021 26797 tablet_service.cc:1467] Processing CreateTablet for tablet ce03ee354c54457487ea939301ccd90e (DEFAULT_TABLE table=rebalance_test_table_17 [id=796786e80d834485b2cee5a25161aa97]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:42.121284 26406 tablet_service.cc:1467] Processing CreateTablet for tablet ce03ee354c54457487ea939301ccd90e (DEFAULT_TABLE table=rebalance_test_table_17 [id=796786e80d834485b2cee5a25161aa97]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:42.121553 26797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce03ee354c54457487ea939301ccd90e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:42.121805 26406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce03ee354c54457487ea939301ccd90e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:42.123669 26667 tablet_service.cc:1467] Processing CreateTablet for tablet ce03ee354c54457487ea939301ccd90e (DEFAULT_TABLE table=rebalance_test_table_17 [id=796786e80d834485b2cee5a25161aa97]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:42.124147 26667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce03ee354c54457487ea939301ccd90e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:42.129997 26537 tablet_service.cc:1467] Processing CreateTablet for tablet 058eec4efd564c348684857372c60127 (DEFAULT_TABLE table=rebalance_test_table_17 [id=796786e80d834485b2cee5a25161aa97]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:42.130146 26669 tablet_service.cc:1467] Processing CreateTablet for tablet 40aa23be0caa427e89e8fdaf905dc052 (DEFAULT_TABLE table=rebalance_test_table_17 [id=796786e80d834485b2cee5a25161aa97]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:42.130537 26669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40aa23be0caa427e89e8fdaf905dc052. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:42.130537 26537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 058eec4efd564c348684857372c60127. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:42.131700 26405 tablet_service.cc:1467] Processing CreateTablet for tablet 40aa23be0caa427e89e8fdaf905dc052 (DEFAULT_TABLE table=rebalance_test_table_17 [id=796786e80d834485b2cee5a25161aa97]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:42.132211 26405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40aa23be0caa427e89e8fdaf905dc052. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:42.133088 26798 tablet_service.cc:1467] Processing CreateTablet for tablet 40aa23be0caa427e89e8fdaf905dc052 (DEFAULT_TABLE table=rebalance_test_table_17 [id=796786e80d834485b2cee5a25161aa97]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:42.133530 26798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40aa23be0caa427e89e8fdaf905dc052. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:42.135383 26799 tablet_service.cc:1467] Processing CreateTablet for tablet 058eec4efd564c348684857372c60127 (DEFAULT_TABLE table=rebalance_test_table_17 [id=796786e80d834485b2cee5a25161aa97]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:42.135865 26799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 058eec4efd564c348684857372c60127. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:42.138993 28842 tablet_bootstrap.cc:492] T ce03ee354c54457487ea939301ccd90e P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:42.140952 26407 tablet_service.cc:1467] Processing CreateTablet for tablet 058eec4efd564c348684857372c60127 (DEFAULT_TABLE table=rebalance_test_table_17 [id=796786e80d834485b2cee5a25161aa97]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:42.141427 26407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 058eec4efd564c348684857372c60127. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:42.143926 28842 tablet_bootstrap.cc:654] T ce03ee354c54457487ea939301ccd90e P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:42.146284 28842 tablet_bootstrap.cc:492] T ce03ee354c54457487ea939301ccd90e P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:42.146528 28842 ts_tablet_manager.cc:1367] T ce03ee354c54457487ea939301ccd90e P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.002s
I20240502 14:14:42.147495 28842 raft_consensus.cc:348] T ce03ee354c54457487ea939301ccd90e P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:42.147814 28842 raft_consensus.cc:374] T ce03ee354c54457487ea939301ccd90e P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:42.148272 28842 raft_consensus.cc:729] T ce03ee354c54457487ea939301ccd90e P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:42.148571 28842 consensus_queue.cc:260] T ce03ee354c54457487ea939301ccd90e P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:42.149413 28842 ts_tablet_manager.cc:1398] T ce03ee354c54457487ea939301ccd90e P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:14:42.149905 28842 tablet_bootstrap.cc:492] T 40aa23be0caa427e89e8fdaf905dc052 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:42.151544 28842 tablet_bootstrap.cc:654] T 40aa23be0caa427e89e8fdaf905dc052 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:42.163225 28844 tablet_bootstrap.cc:492] T 058eec4efd564c348684857372c60127 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:42.164459 28846 tablet_bootstrap.cc:492] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:42.167917 28846 tablet_bootstrap.cc:654] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:42.169942 28841 tablet_bootstrap.cc:492] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:42.168329 28844 tablet_bootstrap.cc:654] T 058eec4efd564c348684857372c60127 P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:42.173300 28842 tablet_bootstrap.cc:492] T 40aa23be0caa427e89e8fdaf905dc052 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:42.173539 28842 ts_tablet_manager.cc:1367] T 40aa23be0caa427e89e8fdaf905dc052 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.011s
I20240502 14:14:42.174283 28842 raft_consensus.cc:348] T 40aa23be0caa427e89e8fdaf905dc052 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:42.174544 28842 raft_consensus.cc:374] T 40aa23be0caa427e89e8fdaf905dc052 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:42.174592 28844 tablet_bootstrap.cc:492] T 058eec4efd564c348684857372c60127 P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:42.174640 28842 raft_consensus.cc:729] T 40aa23be0caa427e89e8fdaf905dc052 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:42.174747 28841 tablet_bootstrap.cc:654] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:42.174780 28844 ts_tablet_manager.cc:1367] T 058eec4efd564c348684857372c60127 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.002s
I20240502 14:14:42.174949 28842 consensus_queue.cc:260] T 40aa23be0caa427e89e8fdaf905dc052 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:42.175659 28842 ts_tablet_manager.cc:1398] T 40aa23be0caa427e89e8fdaf905dc052 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:14:42.175712 28844 raft_consensus.cc:348] T 058eec4efd564c348684857372c60127 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:42.176016 28844 raft_consensus.cc:374] T 058eec4efd564c348684857372c60127 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:42.176179 28844 raft_consensus.cc:729] T 058eec4efd564c348684857372c60127 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:42.176225 28842 tablet_bootstrap.cc:492] T 058eec4efd564c348684857372c60127 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:42.176617 28846 tablet_bootstrap.cc:492] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:42.176623 28844 consensus_queue.cc:260] T 058eec4efd564c348684857372c60127 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:42.176860 28846 ts_tablet_manager.cc:1367] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.008s
I20240502 14:14:42.177949 28846 raft_consensus.cc:348] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:42.178279 28846 raft_consensus.cc:374] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:42.178431 28846 raft_consensus.cc:729] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:42.178598 28844 ts_tablet_manager.cc:1398] T 058eec4efd564c348684857372c60127 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 14:14:42.178844 28846 consensus_queue.cc:260] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:42.179807 28846 ts_tablet_manager.cc:1398] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:14:42.180974 28846 tablet_bootstrap.cc:492] T 40aa23be0caa427e89e8fdaf905dc052 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:42.181672 28842 tablet_bootstrap.cc:654] T 058eec4efd564c348684857372c60127 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:42.182833 28846 tablet_bootstrap.cc:654] T 40aa23be0caa427e89e8fdaf905dc052 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:42.186213 28841 tablet_bootstrap.cc:492] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:42.186518 28841 ts_tablet_manager.cc:1367] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240502 14:14:42.187520 28841 raft_consensus.cc:348] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:42.187881 28841 raft_consensus.cc:374] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:42.188053 28841 raft_consensus.cc:729] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:42.188477 28841 consensus_queue.cc:260] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:42.190431 28841 ts_tablet_manager.cc:1398] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:14:42.190851 28841 tablet_bootstrap.cc:492] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:42.192942 28841 tablet_bootstrap.cc:654] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:42.195788 28842 tablet_bootstrap.cc:492] T 058eec4efd564c348684857372c60127 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:42.195993 28842 ts_tablet_manager.cc:1367] T 058eec4efd564c348684857372c60127 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.000s
I20240502 14:14:42.196372 28846 tablet_bootstrap.cc:492] T 40aa23be0caa427e89e8fdaf905dc052 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:42.196651 28846 ts_tablet_manager.cc:1367] T 40aa23be0caa427e89e8fdaf905dc052 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.003s
I20240502 14:14:42.197003 28842 raft_consensus.cc:348] T 058eec4efd564c348684857372c60127 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:42.197188 28842 raft_consensus.cc:374] T 058eec4efd564c348684857372c60127 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:42.197301 28842 raft_consensus.cc:729] T 058eec4efd564c348684857372c60127 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:42.197535 28846 raft_consensus.cc:348] T 40aa23be0caa427e89e8fdaf905dc052 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:42.197574 28842 consensus_queue.cc:260] T 058eec4efd564c348684857372c60127 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:42.197796 28846 raft_consensus.cc:374] T 40aa23be0caa427e89e8fdaf905dc052 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:42.197919 28846 raft_consensus.cc:729] T 40aa23be0caa427e89e8fdaf905dc052 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:42.198220 28846 consensus_queue.cc:260] T 40aa23be0caa427e89e8fdaf905dc052 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:42.198704 28846 ts_tablet_manager.cc:1398] T 40aa23be0caa427e89e8fdaf905dc052 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:14:42.199163 28842 ts_tablet_manager.cc:1398] T 058eec4efd564c348684857372c60127 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:42.201794 28841 tablet_bootstrap.cc:492] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:42.202011 28841 ts_tablet_manager.cc:1367] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240502 14:14:42.202919 28841 raft_consensus.cc:348] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:42.203122 28841 raft_consensus.cc:374] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:42.203214 28841 raft_consensus.cc:729] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:42.203475 28841 consensus_queue.cc:260] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:42.204384 28841 ts_tablet_manager.cc:1398] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:42.204960 28841 tablet_bootstrap.cc:492] T 058eec4efd564c348684857372c60127 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:42.206848 28841 tablet_bootstrap.cc:654] T 058eec4efd564c348684857372c60127 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:42.209353 28841 tablet_bootstrap.cc:492] T 058eec4efd564c348684857372c60127 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:42.209627 28841 ts_tablet_manager.cc:1367] T 058eec4efd564c348684857372c60127 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 14:14:42.212004 28841 raft_consensus.cc:348] T 058eec4efd564c348684857372c60127 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:42.212255 28841 raft_consensus.cc:374] T 058eec4efd564c348684857372c60127 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:42.212419 28841 raft_consensus.cc:729] T 058eec4efd564c348684857372c60127 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:42.212749 28841 consensus_queue.cc:260] T 058eec4efd564c348684857372c60127 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:42.214000 28841 ts_tablet_manager.cc:1398] T 058eec4efd564c348684857372c60127 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:14:42.279138 28590 raft_consensus.cc:482] T 058eec4efd564c348684857372c60127 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:42.279325 28590 raft_consensus.cc:504] T 058eec4efd564c348684857372c60127 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:42.280030 28590 leader_election.cc:288] T 058eec4efd564c348684857372c60127 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:42.280594 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "058eec4efd564c348684857372c60127" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:42.280699 26817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "058eec4efd564c348684857372c60127" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:14:42.280937 26817 raft_consensus.cc:2454] T 058eec4efd564c348684857372c60127 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:14:42.280938 26558 raft_consensus.cc:2454] T 058eec4efd564c348684857372c60127 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:14:42.281518 26363 leader_election.cc:302] T 058eec4efd564c348684857372c60127 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:42.281881 28590 raft_consensus.cc:2781] T 058eec4efd564c348684857372c60127 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:42.282014 28590 raft_consensus.cc:482] T 058eec4efd564c348684857372c60127 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:42.282120 28590 raft_consensus.cc:3037] T 058eec4efd564c348684857372c60127 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:42.284186 28590 raft_consensus.cc:504] T 058eec4efd564c348684857372c60127 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:42.284886 28590 leader_election.cc:288] T 058eec4efd564c348684857372c60127 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Requested vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:42.285543 26817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "058eec4efd564c348684857372c60127" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:14:42.285782 26817 raft_consensus.cc:3037] T 058eec4efd564c348684857372c60127 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:42.285795 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "058eec4efd564c348684857372c60127" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:14:42.286028 26558 raft_consensus.cc:3037] T 058eec4efd564c348684857372c60127 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:42.287861 26817 raft_consensus.cc:2454] T 058eec4efd564c348684857372c60127 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:14:42.288031 26558 raft_consensus.cc:2454] T 058eec4efd564c348684857372c60127 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:14:42.288475 26363 leader_election.cc:302] T 058eec4efd564c348684857372c60127 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:42.288851 28590 raft_consensus.cc:2781] T 058eec4efd564c348684857372c60127 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:42.288991 28590 raft_consensus.cc:686] T 058eec4efd564c348684857372c60127 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Becoming Leader. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Running, Role: LEADER
I20240502 14:14:42.289320 28590 consensus_queue.cc:237] T 058eec4efd564c348684857372c60127 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:42.293378 26280 catalog_manager.cc:5484] T 058eec4efd564c348684857372c60127 P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: term changed from 0 to 1, leader changed from <none> to 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193). New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:14:42.336181 28744 raft_consensus.cc:482] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:42.336433 28744 raft_consensus.cc:504] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:42.337396 28744 leader_election.cc:288] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:14:42.338064 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce03ee354c54457487ea939301ccd90e" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:14:42.338034 26689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce03ee354c54457487ea939301ccd90e" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:14:42.338333 26689 raft_consensus.cc:2454] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 0.
I20240502 14:14:42.338353 26426 raft_consensus.cc:2454] T ce03ee354c54457487ea939301ccd90e P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 0.
I20240502 14:14:42.338901 26754 leader_election.cc:302] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:42.339592 28744 raft_consensus.cc:2781] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:42.340420 28744 raft_consensus.cc:482] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:42.340574 28744 raft_consensus.cc:3037] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:42.342589 28744 raft_consensus.cc:504] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:42.343178 28744 leader_election.cc:288] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Requested vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:14:42.343822 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce03ee354c54457487ea939301ccd90e" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:14:42.343849 26687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce03ee354c54457487ea939301ccd90e" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:14:42.344036 26426 raft_consensus.cc:3037] T ce03ee354c54457487ea939301ccd90e P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:42.344070 26687 raft_consensus.cc:3037] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:42.346669 26687 raft_consensus.cc:2454] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
I20240502 14:14:42.347142 26426 raft_consensus.cc:2454] T ce03ee354c54457487ea939301ccd90e P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
I20240502 14:14:42.347254 26755 leader_election.cc:302] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:42.347844 28744 raft_consensus.cc:2781] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:42.347986 28744 raft_consensus.cc:686] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Becoming Leader. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Running, Role: LEADER
I20240502 14:14:42.348361 28744 consensus_queue.cc:237] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:42.352229 26280 catalog_manager.cc:5484] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196). New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:14:42.370590 28744 raft_consensus.cc:482] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:42.370848 28744 raft_consensus.cc:504] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:42.371563 28744 leader_election.cc:288] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:14:42.372593 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40aa23be0caa427e89e8fdaf905dc052" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:14:42.372610 26687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40aa23be0caa427e89e8fdaf905dc052" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:14:42.372903 26426 raft_consensus.cc:2454] T 40aa23be0caa427e89e8fdaf905dc052 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 0.
I20240502 14:14:42.373040 26687 raft_consensus.cc:2454] T 40aa23be0caa427e89e8fdaf905dc052 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 0.
I20240502 14:14:42.373445 26754 leader_election.cc:302] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:42.373824 28744 raft_consensus.cc:2781] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:42.373912 28744 raft_consensus.cc:482] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:42.373991 28744 raft_consensus.cc:3037] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:42.375941 28744 raft_consensus.cc:504] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:42.376607 28744 leader_election.cc:288] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Requested vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:14:42.377314 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40aa23be0caa427e89e8fdaf905dc052" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:14:42.377283 26687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40aa23be0caa427e89e8fdaf905dc052" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:14:42.377529 26426 raft_consensus.cc:3037] T 40aa23be0caa427e89e8fdaf905dc052 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:42.377547 26687 raft_consensus.cc:3037] T 40aa23be0caa427e89e8fdaf905dc052 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:42.380326 26687 raft_consensus.cc:2454] T 40aa23be0caa427e89e8fdaf905dc052 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
I20240502 14:14:42.380411 26426 raft_consensus.cc:2454] T 40aa23be0caa427e89e8fdaf905dc052 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
I20240502 14:14:42.380836 26754 leader_election.cc:302] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:42.381285 28744 raft_consensus.cc:2781] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:42.381495 28744 raft_consensus.cc:686] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Becoming Leader. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Running, Role: LEADER
I20240502 14:14:42.381837 28744 consensus_queue.cc:237] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:42.386006 26280 catalog_manager.cc:5484] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196). New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:14:42.715994 28590 consensus_queue.cc:1035] T 058eec4efd564c348684857372c60127 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:42.747732 28590 consensus_queue.cc:1035] T 058eec4efd564c348684857372c60127 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:42.833850 28748 consensus_queue.cc:1035] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:42.842375 28744 consensus_queue.cc:1035] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:42.860889 28858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:42.861280 28858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 14:14:42.868978 28744 consensus_queue.cc:1035] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:42.870600 28858 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:14:42.875198 28872 consensus_queue.cc:1035] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_17'
INSERT report
rows total: 2000
time total: 274.487 ms
time per row: 0.137243 ms
I20240502 14:14:43.315011 26276 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalance_test_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:43.325820 26537 tablet_service.cc:1467] Processing CreateTablet for tablet 001f99a64fb94ec4ba7abacc41953cb7 (DEFAULT_TABLE table=rebalance_test_table_18 [id=5b975fa2c29340a4a7018f3ddff103a0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:43.326023 26407 tablet_service.cc:1467] Processing CreateTablet for tablet 001f99a64fb94ec4ba7abacc41953cb7 (DEFAULT_TABLE table=rebalance_test_table_18 [id=5b975fa2c29340a4a7018f3ddff103a0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:43.326264 26667 tablet_service.cc:1467] Processing CreateTablet for tablet 001f99a64fb94ec4ba7abacc41953cb7 (DEFAULT_TABLE table=rebalance_test_table_18 [id=5b975fa2c29340a4a7018f3ddff103a0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:43.326391 26537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 001f99a64fb94ec4ba7abacc41953cb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:43.326481 26407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 001f99a64fb94ec4ba7abacc41953cb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:43.326710 26667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 001f99a64fb94ec4ba7abacc41953cb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:43.326391 26538 tablet_service.cc:1467] Processing CreateTablet for tablet d1b08a572d684a82b85c5f9f461435ab (DEFAULT_TABLE table=rebalance_test_table_18 [id=5b975fa2c29340a4a7018f3ddff103a0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:43.328842 26536 tablet_service.cc:1467] Processing CreateTablet for tablet f7773bd35a1941c690e620753f42019b (DEFAULT_TABLE table=rebalance_test_table_18 [id=5b975fa2c29340a4a7018f3ddff103a0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:43.329181 26538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1b08a572d684a82b85c5f9f461435ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:43.329221 26797 tablet_service.cc:1467] Processing CreateTablet for tablet d1b08a572d684a82b85c5f9f461435ab (DEFAULT_TABLE table=rebalance_test_table_18 [id=5b975fa2c29340a4a7018f3ddff103a0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:43.329648 26797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1b08a572d684a82b85c5f9f461435ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:43.330292 26406 tablet_service.cc:1467] Processing CreateTablet for tablet f7773bd35a1941c690e620753f42019b (DEFAULT_TABLE table=rebalance_test_table_18 [id=5b975fa2c29340a4a7018f3ddff103a0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:43.330705 26406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7773bd35a1941c690e620753f42019b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:43.332863 26536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7773bd35a1941c690e620753f42019b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:43.336227 26669 tablet_service.cc:1467] Processing CreateTablet for tablet d1b08a572d684a82b85c5f9f461435ab (DEFAULT_TABLE table=rebalance_test_table_18 [id=5b975fa2c29340a4a7018f3ddff103a0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:43.337069 26669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1b08a572d684a82b85c5f9f461435ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:43.340929 26668 tablet_service.cc:1467] Processing CreateTablet for tablet f7773bd35a1941c690e620753f42019b (DEFAULT_TABLE table=rebalance_test_table_18 [id=5b975fa2c29340a4a7018f3ddff103a0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:43.341420 26668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7773bd35a1941c690e620753f42019b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:43.346649 28927 tablet_bootstrap.cc:492] T 001f99a64fb94ec4ba7abacc41953cb7 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:43.348608 28927 tablet_bootstrap.cc:654] T 001f99a64fb94ec4ba7abacc41953cb7 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:43.374373 28928 tablet_bootstrap.cc:492] T 001f99a64fb94ec4ba7abacc41953cb7 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:43.380661 28929 tablet_bootstrap.cc:492] T d1b08a572d684a82b85c5f9f461435ab P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:43.381390 28926 tablet_bootstrap.cc:492] T d1b08a572d684a82b85c5f9f461435ab P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:43.383015 28927 tablet_bootstrap.cc:492] T 001f99a64fb94ec4ba7abacc41953cb7 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:43.383284 28927 ts_tablet_manager.cc:1367] T 001f99a64fb94ec4ba7abacc41953cb7 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.037s user 0.006s sys 0.010s
I20240502 14:14:43.383464 28929 tablet_bootstrap.cc:654] T d1b08a572d684a82b85c5f9f461435ab P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:43.383669 28926 tablet_bootstrap.cc:654] T d1b08a572d684a82b85c5f9f461435ab P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:43.384099 28927 raft_consensus.cc:348] T 001f99a64fb94ec4ba7abacc41953cb7 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:43.384347 28927 raft_consensus.cc:374] T 001f99a64fb94ec4ba7abacc41953cb7 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:43.384471 28927 raft_consensus.cc:729] T 001f99a64fb94ec4ba7abacc41953cb7 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:43.384770 28927 consensus_queue.cc:260] T 001f99a64fb94ec4ba7abacc41953cb7 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:43.385380 28927 ts_tablet_manager.cc:1398] T 001f99a64fb94ec4ba7abacc41953cb7 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:43.385550 28928 tablet_bootstrap.cc:654] T 001f99a64fb94ec4ba7abacc41953cb7 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:43.386744 28927 tablet_bootstrap.cc:492] T f7773bd35a1941c690e620753f42019b P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:43.388779 28927 tablet_bootstrap.cc:654] T f7773bd35a1941c690e620753f42019b P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:43.394860 28926 tablet_bootstrap.cc:492] T d1b08a572d684a82b85c5f9f461435ab P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:43.395058 28928 tablet_bootstrap.cc:492] T 001f99a64fb94ec4ba7abacc41953cb7 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:43.395085 28926 ts_tablet_manager.cc:1367] T d1b08a572d684a82b85c5f9f461435ab P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20240502 14:14:43.395246 28928 ts_tablet_manager.cc:1367] T 001f99a64fb94ec4ba7abacc41953cb7 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.000s
I20240502 14:14:43.395953 28928 raft_consensus.cc:348] T 001f99a64fb94ec4ba7abacc41953cb7 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:43.396035 28926 raft_consensus.cc:348] T d1b08a572d684a82b85c5f9f461435ab P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:43.396242 28928 raft_consensus.cc:374] T 001f99a64fb94ec4ba7abacc41953cb7 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:43.396288 28929 tablet_bootstrap.cc:492] T d1b08a572d684a82b85c5f9f461435ab P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:43.396291 28926 raft_consensus.cc:374] T d1b08a572d684a82b85c5f9f461435ab P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:43.396417 28926 raft_consensus.cc:729] T d1b08a572d684a82b85c5f9f461435ab P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:43.396494 28929 ts_tablet_manager.cc:1367] T d1b08a572d684a82b85c5f9f461435ab P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.005s
I20240502 14:14:43.396710 28928 raft_consensus.cc:729] T 001f99a64fb94ec4ba7abacc41953cb7 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:43.396706 28926 consensus_queue.cc:260] T d1b08a572d684a82b85c5f9f461435ab P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:43.397032 28928 consensus_queue.cc:260] T 001f99a64fb94ec4ba7abacc41953cb7 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:43.397363 28926 ts_tablet_manager.cc:1398] T d1b08a572d684a82b85c5f9f461435ab P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:14:43.397361 28929 raft_consensus.cc:348] T d1b08a572d684a82b85c5f9f461435ab P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:43.397616 28929 raft_consensus.cc:374] T d1b08a572d684a82b85c5f9f461435ab P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:43.397744 28926 tablet_bootstrap.cc:492] T 001f99a64fb94ec4ba7abacc41953cb7 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:43.397742 28929 raft_consensus.cc:729] T d1b08a572d684a82b85c5f9f461435ab P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:43.398118 28929 consensus_queue.cc:260] T d1b08a572d684a82b85c5f9f461435ab P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:43.399590 28926 tablet_bootstrap.cc:654] T 001f99a64fb94ec4ba7abacc41953cb7 P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:43.399716 28928 ts_tablet_manager.cc:1398] T 001f99a64fb94ec4ba7abacc41953cb7 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:43.400058 28928 tablet_bootstrap.cc:492] T f7773bd35a1941c690e620753f42019b P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:43.400592 28927 tablet_bootstrap.cc:492] T f7773bd35a1941c690e620753f42019b P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:43.401783 28927 ts_tablet_manager.cc:1367] T f7773bd35a1941c690e620753f42019b P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.000s
I20240502 14:14:43.402102 28928 tablet_bootstrap.cc:654] T f7773bd35a1941c690e620753f42019b P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:43.403399 28929 ts_tablet_manager.cc:1398] T d1b08a572d684a82b85c5f9f461435ab P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:14:43.403708 28927 raft_consensus.cc:348] T f7773bd35a1941c690e620753f42019b P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:43.404017 28927 raft_consensus.cc:374] T f7773bd35a1941c690e620753f42019b P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:43.404119 28927 raft_consensus.cc:729] T f7773bd35a1941c690e620753f42019b P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:43.404472 28927 consensus_queue.cc:260] T f7773bd35a1941c690e620753f42019b P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:43.406872 28927 ts_tablet_manager.cc:1398] T f7773bd35a1941c690e620753f42019b P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240502 14:14:43.407328 28927 tablet_bootstrap.cc:492] T d1b08a572d684a82b85c5f9f461435ab P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:43.409699 28927 tablet_bootstrap.cc:654] T d1b08a572d684a82b85c5f9f461435ab P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:43.412071 28926 tablet_bootstrap.cc:492] T 001f99a64fb94ec4ba7abacc41953cb7 P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:43.412375 28926 ts_tablet_manager.cc:1367] T 001f99a64fb94ec4ba7abacc41953cb7 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.010s
I20240502 14:14:43.413571 28926 raft_consensus.cc:348] T 001f99a64fb94ec4ba7abacc41953cb7 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:43.413937 28926 raft_consensus.cc:374] T 001f99a64fb94ec4ba7abacc41953cb7 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:43.414050 28926 raft_consensus.cc:729] T 001f99a64fb94ec4ba7abacc41953cb7 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:43.414361 28926 consensus_queue.cc:260] T 001f99a64fb94ec4ba7abacc41953cb7 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:43.415088 28926 ts_tablet_manager.cc:1398] T 001f99a64fb94ec4ba7abacc41953cb7 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:14:43.415499 28926 tablet_bootstrap.cc:492] T f7773bd35a1941c690e620753f42019b P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:43.415760 28928 tablet_bootstrap.cc:492] T f7773bd35a1941c690e620753f42019b P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:43.416599 28927 tablet_bootstrap.cc:492] T d1b08a572d684a82b85c5f9f461435ab P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:43.416821 28927 ts_tablet_manager.cc:1367] T d1b08a572d684a82b85c5f9f461435ab P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.005s
I20240502 14:14:43.417850 28927 raft_consensus.cc:348] T d1b08a572d684a82b85c5f9f461435ab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:43.418133 28927 raft_consensus.cc:374] T d1b08a572d684a82b85c5f9f461435ab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:43.418272 28927 raft_consensus.cc:729] T d1b08a572d684a82b85c5f9f461435ab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:43.418570 28927 consensus_queue.cc:260] T d1b08a572d684a82b85c5f9f461435ab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:43.419248 28926 tablet_bootstrap.cc:654] T f7773bd35a1941c690e620753f42019b P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:43.419248 28927 ts_tablet_manager.cc:1398] T d1b08a572d684a82b85c5f9f461435ab P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:43.419633 28928 ts_tablet_manager.cc:1367] T f7773bd35a1941c690e620753f42019b P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.011s
I20240502 14:14:43.420539 28928 raft_consensus.cc:348] T f7773bd35a1941c690e620753f42019b P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:43.421967 28928 raft_consensus.cc:374] T f7773bd35a1941c690e620753f42019b P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:43.422096 28928 raft_consensus.cc:729] T f7773bd35a1941c690e620753f42019b P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:43.422434 28928 consensus_queue.cc:260] T f7773bd35a1941c690e620753f42019b P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:43.423053 28928 ts_tablet_manager.cc:1398] T f7773bd35a1941c690e620753f42019b P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:43.423246 28926 tablet_bootstrap.cc:492] T f7773bd35a1941c690e620753f42019b P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:43.423434 28926 ts_tablet_manager.cc:1367] T f7773bd35a1941c690e620753f42019b P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20240502 14:14:43.424290 28926 raft_consensus.cc:348] T f7773bd35a1941c690e620753f42019b P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:43.424561 28926 raft_consensus.cc:374] T f7773bd35a1941c690e620753f42019b P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:43.424692 28926 raft_consensus.cc:729] T f7773bd35a1941c690e620753f42019b P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:43.424969 28926 consensus_queue.cc:260] T f7773bd35a1941c690e620753f42019b P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:43.425592 28926 ts_tablet_manager.cc:1398] T f7773bd35a1941c690e620753f42019b P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:43.444590 28809 raft_consensus.cc:482] T f7773bd35a1941c690e620753f42019b P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:43.444831 28809 raft_consensus.cc:504] T f7773bd35a1941c690e620753f42019b P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:43.445621 28809 leader_election.cc:288] T f7773bd35a1941c690e620753f42019b P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:14:43.445976 26424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7773bd35a1941c690e620753f42019b" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:14:43.446156 26556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7773bd35a1941c690e620753f42019b" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:43.446233 26424 raft_consensus.cc:2454] T f7773bd35a1941c690e620753f42019b P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:14:43.446437 26556 raft_consensus.cc:2454] T f7773bd35a1941c690e620753f42019b P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:14:43.446841 26624 leader_election.cc:302] T f7773bd35a1941c690e620753f42019b P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:14:43.447161 28809 raft_consensus.cc:2781] T f7773bd35a1941c690e620753f42019b P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:43.447273 28809 raft_consensus.cc:482] T f7773bd35a1941c690e620753f42019b P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:43.447383 28809 raft_consensus.cc:3037] T f7773bd35a1941c690e620753f42019b P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:43.449342 28809 raft_consensus.cc:504] T f7773bd35a1941c690e620753f42019b P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:43.449901 28809 leader_election.cc:288] T f7773bd35a1941c690e620753f42019b P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Requested vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:14:43.450338 26424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7773bd35a1941c690e620753f42019b" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:14:43.450444 26556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7773bd35a1941c690e620753f42019b" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:14:43.450595 26424 raft_consensus.cc:3037] T f7773bd35a1941c690e620753f42019b P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:43.450678 26556 raft_consensus.cc:3037] T f7773bd35a1941c690e620753f42019b P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:43.452984 26424 raft_consensus.cc:2454] T f7773bd35a1941c690e620753f42019b P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:14:43.452983 26556 raft_consensus.cc:2454] T f7773bd35a1941c690e620753f42019b P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:14:43.453493 26624 leader_election.cc:302] T f7773bd35a1941c690e620753f42019b P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:14:43.453847 28809 raft_consensus.cc:2781] T f7773bd35a1941c690e620753f42019b P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:43.454042 28809 raft_consensus.cc:686] T f7773bd35a1941c690e620753f42019b P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Becoming Leader. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Running, Role: LEADER
I20240502 14:14:43.454581 28809 consensus_queue.cc:237] T f7773bd35a1941c690e620753f42019b P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:43.458531 26277 catalog_manager.cc:5484] T f7773bd35a1941c690e620753f42019b P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195). New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:14:43.460031 28778 raft_consensus.cc:482] T 001f99a64fb94ec4ba7abacc41953cb7 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:43.460215 28778 raft_consensus.cc:504] T 001f99a64fb94ec4ba7abacc41953cb7 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:43.460981 28778 leader_election.cc:288] T 001f99a64fb94ec4ba7abacc41953cb7 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:14:43.461436 26689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001f99a64fb94ec4ba7abacc41953cb7" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:14:43.461692 26689 raft_consensus.cc:2454] T 001f99a64fb94ec4ba7abacc41953cb7 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 0.
I20240502 14:14:43.462091 26424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001f99a64fb94ec4ba7abacc41953cb7" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:14:43.462203 26494 leader_election.cc:302] T 001f99a64fb94ec4ba7abacc41953cb7 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:43.462323 26424 raft_consensus.cc:2454] T 001f99a64fb94ec4ba7abacc41953cb7 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 0.
I20240502 14:14:43.462509 28778 raft_consensus.cc:2781] T 001f99a64fb94ec4ba7abacc41953cb7 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:43.462635 28778 raft_consensus.cc:482] T 001f99a64fb94ec4ba7abacc41953cb7 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:43.462735 28778 raft_consensus.cc:3037] T 001f99a64fb94ec4ba7abacc41953cb7 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:43.464884 28778 raft_consensus.cc:504] T 001f99a64fb94ec4ba7abacc41953cb7 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:43.466459 28778 leader_election.cc:288] T 001f99a64fb94ec4ba7abacc41953cb7 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Requested vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:14:43.467361 26687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001f99a64fb94ec4ba7abacc41953cb7" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:14:43.467623 26687 raft_consensus.cc:3037] T 001f99a64fb94ec4ba7abacc41953cb7 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:43.467574 26424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001f99a64fb94ec4ba7abacc41953cb7" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:14:43.468122 26424 raft_consensus.cc:3037] T 001f99a64fb94ec4ba7abacc41953cb7 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:43.470793 26687 raft_consensus.cc:2454] T 001f99a64fb94ec4ba7abacc41953cb7 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:14:43.471149 26424 raft_consensus.cc:2454] T 001f99a64fb94ec4ba7abacc41953cb7 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:14:43.471355 26494 leader_election.cc:302] T 001f99a64fb94ec4ba7abacc41953cb7 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:43.471721 28778 raft_consensus.cc:2781] T 001f99a64fb94ec4ba7abacc41953cb7 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:43.471858 28778 raft_consensus.cc:686] T 001f99a64fb94ec4ba7abacc41953cb7 P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Becoming Leader. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Running, Role: LEADER
I20240502 14:14:43.472141 28778 consensus_queue.cc:237] T 001f99a64fb94ec4ba7abacc41953cb7 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:43.476720 26277 catalog_manager.cc:5484] T 001f99a64fb94ec4ba7abacc41953cb7 P ba187e38a2754d60a12de9ac4856430a reported cstate change: term changed from 0 to 1, leader changed from <none> to ba187e38a2754d60a12de9ac4856430a (127.19.94.194). New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:14:43.634953 28809 raft_consensus.cc:482] T d1b08a572d684a82b85c5f9f461435ab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:43.635151 28809 raft_consensus.cc:504] T d1b08a572d684a82b85c5f9f461435ab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:43.636004 28809 leader_election.cc:288] T d1b08a572d684a82b85c5f9f461435ab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:43.636649 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1b08a572d684a82b85c5f9f461435ab" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:43.636698 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1b08a572d684a82b85c5f9f461435ab" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:14:43.636909 26558 raft_consensus.cc:2454] T d1b08a572d684a82b85c5f9f461435ab P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:14:43.636915 26818 raft_consensus.cc:2454] T d1b08a572d684a82b85c5f9f461435ab P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:14:43.637478 26625 leader_election.cc:302] T d1b08a572d684a82b85c5f9f461435ab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:43.637804 28809 raft_consensus.cc:2781] T d1b08a572d684a82b85c5f9f461435ab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:43.637944 28809 raft_consensus.cc:482] T d1b08a572d684a82b85c5f9f461435ab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:43.638065 28809 raft_consensus.cc:3037] T d1b08a572d684a82b85c5f9f461435ab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:43.640333 28809 raft_consensus.cc:504] T d1b08a572d684a82b85c5f9f461435ab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:43.641017 28809 leader_election.cc:288] T d1b08a572d684a82b85c5f9f461435ab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Requested vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:43.641530 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1b08a572d684a82b85c5f9f461435ab" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:14:43.641606 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1b08a572d684a82b85c5f9f461435ab" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:14:43.641736 26558 raft_consensus.cc:3037] T d1b08a572d684a82b85c5f9f461435ab P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:43.641829 26818 raft_consensus.cc:3037] T d1b08a572d684a82b85c5f9f461435ab P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:43.643657 26558 raft_consensus.cc:2454] T d1b08a572d684a82b85c5f9f461435ab P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:14:43.643723 26818 raft_consensus.cc:2454] T d1b08a572d684a82b85c5f9f461435ab P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:14:43.644189 26624 leader_election.cc:302] T d1b08a572d684a82b85c5f9f461435ab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:43.644508 28809 raft_consensus.cc:2781] T d1b08a572d684a82b85c5f9f461435ab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:43.644657 28809 raft_consensus.cc:686] T d1b08a572d684a82b85c5f9f461435ab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Becoming Leader. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Running, Role: LEADER
I20240502 14:14:43.645001 28809 consensus_queue.cc:237] T d1b08a572d684a82b85c5f9f461435ab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:43.648487 26277 catalog_manager.cc:5484] T d1b08a572d684a82b85c5f9f461435ab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195). New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:14:43.870307 28809 consensus_queue.cc:1035] T f7773bd35a1941c690e620753f42019b P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:43.877825 28809 consensus_queue.cc:1035] T f7773bd35a1941c690e620753f42019b P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:43.920679 28860 consensus_queue.cc:1035] T 001f99a64fb94ec4ba7abacc41953cb7 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:43.925619 28860 consensus_queue.cc:1035] T 001f99a64fb94ec4ba7abacc41953cb7 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:44.055640 28943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:44.056020 28943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:44.064644 28943 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:14:44.094492 28930 consensus_queue.cc:1035] T d1b08a572d684a82b85c5f9f461435ab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:44.099673 28946 consensus_queue.cc:1035] T d1b08a572d684a82b85c5f9f461435ab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_18'
W20240502 14:14:44.320463 26494 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 84524.4us
INSERT report
rows total: 2000
time total: 308.1 ms
time per row: 0.15405 ms
I20240502 14:14:44.542703 26277 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalance_test_table_19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:44.552981 26797 tablet_service.cc:1467] Processing CreateTablet for tablet bb3e4ec75888419aa4e065cf5513ae5b (DEFAULT_TABLE table=rebalance_test_table_19 [id=d4a5cfed144a42db8322f6823ef341a8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:44.553525 26797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb3e4ec75888419aa4e065cf5513ae5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:44.553609 26538 tablet_service.cc:1467] Processing CreateTablet for tablet bb3e4ec75888419aa4e065cf5513ae5b (DEFAULT_TABLE table=rebalance_test_table_19 [id=d4a5cfed144a42db8322f6823ef341a8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:44.554064 26538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb3e4ec75888419aa4e065cf5513ae5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:44.556799 26798 tablet_service.cc:1467] Processing CreateTablet for tablet 981e25d29abc44299dc115d6df835c58 (DEFAULT_TABLE table=rebalance_test_table_19 [id=d4a5cfed144a42db8322f6823ef341a8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:44.556818 26537 tablet_service.cc:1467] Processing CreateTablet for tablet cf52d0f868fa45418af48639e12f3282 (DEFAULT_TABLE table=rebalance_test_table_19 [id=d4a5cfed144a42db8322f6823ef341a8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:44.557143 26798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 981e25d29abc44299dc115d6df835c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:44.557219 26537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf52d0f868fa45418af48639e12f3282. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:44.559437 26799 tablet_service.cc:1467] Processing CreateTablet for tablet cf52d0f868fa45418af48639e12f3282 (DEFAULT_TABLE table=rebalance_test_table_19 [id=d4a5cfed144a42db8322f6823ef341a8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:44.559818 26799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf52d0f868fa45418af48639e12f3282. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:44.560849 26407 tablet_service.cc:1467] Processing CreateTablet for tablet bb3e4ec75888419aa4e065cf5513ae5b (DEFAULT_TABLE table=rebalance_test_table_19 [id=d4a5cfed144a42db8322f6823ef341a8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:44.561281 26407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb3e4ec75888419aa4e065cf5513ae5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:44.561971 26669 tablet_service.cc:1467] Processing CreateTablet for tablet cf52d0f868fa45418af48639e12f3282 (DEFAULT_TABLE table=rebalance_test_table_19 [id=d4a5cfed144a42db8322f6823ef341a8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:44.562438 26669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf52d0f868fa45418af48639e12f3282. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:44.565997 26668 tablet_service.cc:1467] Processing CreateTablet for tablet 981e25d29abc44299dc115d6df835c58 (DEFAULT_TABLE table=rebalance_test_table_19 [id=d4a5cfed144a42db8322f6823ef341a8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:44.566442 26668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 981e25d29abc44299dc115d6df835c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:44.567651 29013 tablet_bootstrap.cc:492] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:44.569854 29013 tablet_bootstrap.cc:654] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:44.571697 29012 tablet_bootstrap.cc:492] T bb3e4ec75888419aa4e065cf5513ae5b P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:44.574151 29012 tablet_bootstrap.cc:654] T bb3e4ec75888419aa4e065cf5513ae5b P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:44.580276 29015 tablet_bootstrap.cc:492] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:44.581153 26407 tablet_service.cc:1467] Processing CreateTablet for tablet 981e25d29abc44299dc115d6df835c58 (DEFAULT_TABLE table=rebalance_test_table_19 [id=d4a5cfed144a42db8322f6823ef341a8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:44.581774 26407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 981e25d29abc44299dc115d6df835c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:44.582616 29015 tablet_bootstrap.cc:654] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:44.585469 29013 tablet_bootstrap.cc:492] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:44.585673 29013 ts_tablet_manager.cc:1367] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.000s
I20240502 14:14:44.586593 29013 raft_consensus.cc:348] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:44.586841 29013 raft_consensus.cc:374] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:44.586941 29013 raft_consensus.cc:729] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:44.587155 29013 consensus_queue.cc:260] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:44.587831 29013 ts_tablet_manager.cc:1398] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:44.590260 29013 tablet_bootstrap.cc:492] T 981e25d29abc44299dc115d6df835c58 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:44.592748 29013 tablet_bootstrap.cc:654] T 981e25d29abc44299dc115d6df835c58 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:44.603636 29013 tablet_bootstrap.cc:492] T 981e25d29abc44299dc115d6df835c58 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:44.604128 29013 ts_tablet_manager.cc:1367] T 981e25d29abc44299dc115d6df835c58 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.011s
I20240502 14:14:44.605990 29013 raft_consensus.cc:348] T 981e25d29abc44299dc115d6df835c58 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:44.606462 29013 raft_consensus.cc:374] T 981e25d29abc44299dc115d6df835c58 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:44.606619 29013 raft_consensus.cc:729] T 981e25d29abc44299dc115d6df835c58 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:44.608209 29013 consensus_queue.cc:260] T 981e25d29abc44299dc115d6df835c58 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:44.608753 29003 raft_consensus.cc:482] T 981e25d29abc44299dc115d6df835c58 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:44.608935 29003 raft_consensus.cc:504] T 981e25d29abc44299dc115d6df835c58 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:44.610347 29003 leader_election.cc:288] T 981e25d29abc44299dc115d6df835c58 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:14:44.610982 26817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "981e25d29abc44299dc115d6df835c58" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:14:44.611117 26687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "981e25d29abc44299dc115d6df835c58" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
W20240502 14:14:44.611575 26363 leader_election.cc:341] T 981e25d29abc44299dc115d6df835c58 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:44.611814 29013 ts_tablet_manager.cc:1398] T 981e25d29abc44299dc115d6df835c58 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
W20240502 14:14:44.611981 26363 leader_election.cc:341] T 981e25d29abc44299dc115d6df835c58 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:44.612109 26363 leader_election.cc:302] T 981e25d29abc44299dc115d6df835c58 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d; no voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, e4cfb8fc5a474177af6848a6efa13a27
I20240502 14:14:44.612464 28910 raft_consensus.cc:2726] T 981e25d29abc44299dc115d6df835c58 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:14:44.620786 29019 tablet_bootstrap.cc:492] T cf52d0f868fa45418af48639e12f3282 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:44.623692 29019 tablet_bootstrap.cc:654] T cf52d0f868fa45418af48639e12f3282 P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:44.637177 29015 tablet_bootstrap.cc:492] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:44.637172 29019 tablet_bootstrap.cc:492] T cf52d0f868fa45418af48639e12f3282 P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:44.637354 29015 ts_tablet_manager.cc:1367] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.057s user 0.024s sys 0.018s
I20240502 14:14:44.637601 29019 ts_tablet_manager.cc:1367] T cf52d0f868fa45418af48639e12f3282 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.000s
I20240502 14:14:44.638022 29012 tablet_bootstrap.cc:492] T bb3e4ec75888419aa4e065cf5513ae5b P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:44.638196 29012 ts_tablet_manager.cc:1367] T bb3e4ec75888419aa4e065cf5513ae5b P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.067s user 0.024s sys 0.028s
I20240502 14:14:44.638295 29015 raft_consensus.cc:348] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:44.638602 29015 raft_consensus.cc:374] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:44.638762 29015 raft_consensus.cc:729] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:44.638839 29019 raft_consensus.cc:348] T cf52d0f868fa45418af48639e12f3282 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:44.639030 29012 raft_consensus.cc:348] T bb3e4ec75888419aa4e065cf5513ae5b P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:44.639086 29015 consensus_queue.cc:260] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:44.639236 29012 raft_consensus.cc:374] T bb3e4ec75888419aa4e065cf5513ae5b P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:44.639330 29012 raft_consensus.cc:729] T bb3e4ec75888419aa4e065cf5513ae5b P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:44.639396 29019 raft_consensus.cc:374] T cf52d0f868fa45418af48639e12f3282 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:44.639561 29019 raft_consensus.cc:729] T cf52d0f868fa45418af48639e12f3282 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:44.639855 29019 consensus_queue.cc:260] T cf52d0f868fa45418af48639e12f3282 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:44.640067 29015 ts_tablet_manager.cc:1398] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:14:44.640682 29019 ts_tablet_manager.cc:1398] T cf52d0f868fa45418af48639e12f3282 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:44.640971 29012 consensus_queue.cc:260] T bb3e4ec75888419aa4e065cf5513ae5b P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:44.641518 29019 tablet_bootstrap.cc:492] T bb3e4ec75888419aa4e065cf5513ae5b P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:44.641578 29012 ts_tablet_manager.cc:1398] T bb3e4ec75888419aa4e065cf5513ae5b P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:14:44.641688 29015 tablet_bootstrap.cc:492] T 981e25d29abc44299dc115d6df835c58 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:44.641862 29012 tablet_bootstrap.cc:492] T 981e25d29abc44299dc115d6df835c58 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:44.643465 29015 tablet_bootstrap.cc:654] T 981e25d29abc44299dc115d6df835c58 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:44.645910 29019 tablet_bootstrap.cc:654] T bb3e4ec75888419aa4e065cf5513ae5b P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:44.644925 29012 tablet_bootstrap.cc:654] T 981e25d29abc44299dc115d6df835c58 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:44.651041 29012 tablet_bootstrap.cc:492] T 981e25d29abc44299dc115d6df835c58 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:44.651253 29012 ts_tablet_manager.cc:1367] T 981e25d29abc44299dc115d6df835c58 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.002s
I20240502 14:14:44.652316 29012 raft_consensus.cc:348] T 981e25d29abc44299dc115d6df835c58 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:44.652637 29012 raft_consensus.cc:374] T 981e25d29abc44299dc115d6df835c58 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:44.652746 29012 raft_consensus.cc:729] T 981e25d29abc44299dc115d6df835c58 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:44.653064 29012 consensus_queue.cc:260] T 981e25d29abc44299dc115d6df835c58 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:44.653673 29015 tablet_bootstrap.cc:492] T 981e25d29abc44299dc115d6df835c58 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:44.653846 29012 ts_tablet_manager.cc:1398] T 981e25d29abc44299dc115d6df835c58 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 14:14:44.653973 29015 ts_tablet_manager.cc:1367] T 981e25d29abc44299dc115d6df835c58 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20240502 14:14:44.654145 29012 tablet_bootstrap.cc:492] T cf52d0f868fa45418af48639e12f3282 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:44.654873 29015 raft_consensus.cc:348] T 981e25d29abc44299dc115d6df835c58 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:44.655167 29015 raft_consensus.cc:374] T 981e25d29abc44299dc115d6df835c58 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:44.655292 29015 raft_consensus.cc:729] T 981e25d29abc44299dc115d6df835c58 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:44.655601 29015 consensus_queue.cc:260] T 981e25d29abc44299dc115d6df835c58 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:44.656412 29015 ts_tablet_manager.cc:1398] T 981e25d29abc44299dc115d6df835c58 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:14:44.657402 29012 tablet_bootstrap.cc:654] T cf52d0f868fa45418af48639e12f3282 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:44.662649 29019 tablet_bootstrap.cc:492] T bb3e4ec75888419aa4e065cf5513ae5b P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:44.662860 29019 ts_tablet_manager.cc:1367] T bb3e4ec75888419aa4e065cf5513ae5b P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.007s
I20240502 14:14:44.663733 29019 raft_consensus.cc:348] T bb3e4ec75888419aa4e065cf5513ae5b P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:44.664058 29019 raft_consensus.cc:374] T bb3e4ec75888419aa4e065cf5513ae5b P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:44.664160 29019 raft_consensus.cc:729] T bb3e4ec75888419aa4e065cf5513ae5b P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:44.664484 29019 consensus_queue.cc:260] T bb3e4ec75888419aa4e065cf5513ae5b P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:44.665207 29019 ts_tablet_manager.cc:1398] T bb3e4ec75888419aa4e065cf5513ae5b P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 14:14:44.665773 29012 tablet_bootstrap.cc:492] T cf52d0f868fa45418af48639e12f3282 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:44.665944 29012 ts_tablet_manager.cc:1367] T cf52d0f868fa45418af48639e12f3282 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.004s
I20240502 14:14:44.666682 29012 raft_consensus.cc:348] T cf52d0f868fa45418af48639e12f3282 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:44.666857 29012 raft_consensus.cc:374] T cf52d0f868fa45418af48639e12f3282 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:44.666929 29012 raft_consensus.cc:729] T cf52d0f868fa45418af48639e12f3282 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:44.667137 29012 consensus_queue.cc:260] T cf52d0f868fa45418af48639e12f3282 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:44.667629 29012 ts_tablet_manager.cc:1398] T cf52d0f868fa45418af48639e12f3282 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:14:44.683468 28748 raft_consensus.cc:482] T 981e25d29abc44299dc115d6df835c58 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:44.683709 28748 raft_consensus.cc:504] T 981e25d29abc44299dc115d6df835c58 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:44.684518 28748 leader_election.cc:288] T 981e25d29abc44299dc115d6df835c58 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:14:44.684852 26687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "981e25d29abc44299dc115d6df835c58" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:14:44.685110 26687 raft_consensus.cc:2454] T 981e25d29abc44299dc115d6df835c58 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 0.
I20240502 14:14:44.685123 26423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "981e25d29abc44299dc115d6df835c58" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:14:44.685405 26423 raft_consensus.cc:2454] T 981e25d29abc44299dc115d6df835c58 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 0.
I20240502 14:14:44.685535 26755 leader_election.cc:302] T 981e25d29abc44299dc115d6df835c58 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:44.685828 28748 raft_consensus.cc:2781] T 981e25d29abc44299dc115d6df835c58 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:44.685954 28748 raft_consensus.cc:482] T 981e25d29abc44299dc115d6df835c58 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:44.686056 28748 raft_consensus.cc:3037] T 981e25d29abc44299dc115d6df835c58 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:44.688788 28748 raft_consensus.cc:504] T 981e25d29abc44299dc115d6df835c58 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:44.689738 26687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "981e25d29abc44299dc115d6df835c58" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:14:44.690011 26687 raft_consensus.cc:3037] T 981e25d29abc44299dc115d6df835c58 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:44.690016 28748 leader_election.cc:288] T 981e25d29abc44299dc115d6df835c58 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Requested vote from peers 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:14:44.691977 26687 raft_consensus.cc:2454] T 981e25d29abc44299dc115d6df835c58 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
I20240502 14:14:44.692359 26755 leader_election.cc:302] T 981e25d29abc44299dc115d6df835c58 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:44.692658 28748 raft_consensus.cc:2781] T 981e25d29abc44299dc115d6df835c58 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:44.693259 26423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "981e25d29abc44299dc115d6df835c58" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:14:44.693464 26423 raft_consensus.cc:3037] T 981e25d29abc44299dc115d6df835c58 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:44.697141 26423 raft_consensus.cc:2454] T 981e25d29abc44299dc115d6df835c58 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
I20240502 14:14:44.701942 28748 raft_consensus.cc:686] T 981e25d29abc44299dc115d6df835c58 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Becoming Leader. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Running, Role: LEADER
I20240502 14:14:44.702337 28748 consensus_queue.cc:237] T 981e25d29abc44299dc115d6df835c58 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:44.707255 26278 catalog_manager.cc:5484] T 981e25d29abc44299dc115d6df835c58 P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196). New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:14:44.722110 29004 raft_consensus.cc:482] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:44.722330 29004 raft_consensus.cc:504] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:44.723115 29004 leader_election.cc:288] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:44.723856 26556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf52d0f868fa45418af48639e12f3282" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:44.724110 26556 raft_consensus.cc:2454] T cf52d0f868fa45418af48639e12f3282 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:14:44.724516 26817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf52d0f868fa45418af48639e12f3282" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:14:44.724787 26817 raft_consensus.cc:2454] T cf52d0f868fa45418af48639e12f3282 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:14:44.724898 26624 leader_election.cc:302] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:44.725709 28946 raft_consensus.cc:2781] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:44.725858 28946 raft_consensus.cc:482] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:44.725965 28946 raft_consensus.cc:3037] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:44.729085 28946 raft_consensus.cc:504] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:44.730158 26556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf52d0f868fa45418af48639e12f3282" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:14:44.730393 26556 raft_consensus.cc:3037] T cf52d0f868fa45418af48639e12f3282 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:44.731295 26817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf52d0f868fa45418af48639e12f3282" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:14:44.731509 26817 raft_consensus.cc:3037] T cf52d0f868fa45418af48639e12f3282 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:44.733337 26556 raft_consensus.cc:2454] T cf52d0f868fa45418af48639e12f3282 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:14:44.733973 26624 leader_election.cc:302] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:44.734333 29004 raft_consensus.cc:2781] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:44.734364 26817 raft_consensus.cc:2454] T cf52d0f868fa45418af48639e12f3282 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:14:44.734524 29004 raft_consensus.cc:686] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Becoming Leader. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Running, Role: LEADER
I20240502 14:14:44.734860 29004 consensus_queue.cc:237] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:44.735196 28946 leader_election.cc:288] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Requested vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:44.740882 26278 catalog_manager.cc:5484] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195). New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:14:44.939028 29003 raft_consensus.cc:482] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:44.939258 29003 raft_consensus.cc:504] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:44.940048 29003 leader_election.cc:288] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:14:44.940619 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb3e4ec75888419aa4e065cf5513ae5b" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:14:44.940683 26556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb3e4ec75888419aa4e065cf5513ae5b" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:44.940881 26818 raft_consensus.cc:2454] T bb3e4ec75888419aa4e065cf5513ae5b P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:14:44.940925 26556 raft_consensus.cc:2454] T bb3e4ec75888419aa4e065cf5513ae5b P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:14:44.941366 26363 leader_election.cc:302] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:44.941671 29003 raft_consensus.cc:2781] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:44.941759 29003 raft_consensus.cc:482] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:44.941843 29003 raft_consensus.cc:3037] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:44.943831 29003 raft_consensus.cc:504] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:44.944535 29003 leader_election.cc:288] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Requested vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:14:44.944990 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb3e4ec75888419aa4e065cf5513ae5b" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:14:44.945093 26556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb3e4ec75888419aa4e065cf5513ae5b" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:14:44.945255 26818 raft_consensus.cc:3037] T bb3e4ec75888419aa4e065cf5513ae5b P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:44.945348 26556 raft_consensus.cc:3037] T bb3e4ec75888419aa4e065cf5513ae5b P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:44.948102 26818 raft_consensus.cc:2454] T bb3e4ec75888419aa4e065cf5513ae5b P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:14:44.948102 26556 raft_consensus.cc:2454] T bb3e4ec75888419aa4e065cf5513ae5b P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:14:44.948599 26363 leader_election.cc:302] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:44.948910 29003 raft_consensus.cc:2781] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:44.949070 29003 raft_consensus.cc:686] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Becoming Leader. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Running, Role: LEADER
I20240502 14:14:44.949357 29003 consensus_queue.cc:237] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:44.953291 26278 catalog_manager.cc:5484] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: term changed from 0 to 1, leader changed from <none> to 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193). New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:14:45.111469 28748 consensus_queue.cc:1035] T 981e25d29abc44299dc115d6df835c58 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:45.119284 28748 consensus_queue.cc:1035] T 981e25d29abc44299dc115d6df835c58 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:45.264353 28809 consensus_queue.cc:1035] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:45.276787 28809 consensus_queue.cc:1035] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:45.367530 29029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:45.367923 29029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:45.376317 29029 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:14:45.382061 28910 consensus_queue.cc:1035] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:45.386998 28910 consensus_queue.cc:1035] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_19'
INSERT report
rows total: 2000
time total: 293.894 ms
time per row: 0.146947 ms
I20240502 14:14:45.863088 19835 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_zeJNl/build/asan/bin/kudu
/tmp/dist-test-task_zeJNl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.94.197:38715
--local_ip_for_outbound_sockets=127.19.94.197
--tserver_master_addrs=127.19.94.254:34367
--webserver_port=44271
--webserver_interface=127.19.94.197
--builtin_ntp_servers=127.19.94.212:40605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 14:14:46.280467 29094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:46.280790 29094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:14:46.280892 29094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:46.288746 29094 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:14:46.288942 29094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:14:46.289075 29094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.94.197
I20240502 14:14:46.299808 29094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.94.212:40605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.94.197:38715
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.19.94.197
--webserver_port=44271
--tserver_master_addrs=127.19.94.254:34367
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.94.197
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:14:46.301419 29094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:14:46.304049 29094 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:46.319341 29099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:46.319816 29100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:46.320451 29102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:46.321197 29094 server_base.cc:1034] running on GCE node
I20240502 14:14:46.321909 29094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:14:46.322991 29094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:14:46.324353 29094 hybrid_clock.cc:648] HybridClock initialized: now 1714659286324253 us; error 53 us; skew 500 ppm
I20240502 14:14:46.324939 29094 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:46.327965 29094 webserver.cc:460] Webserver started at http://127.19.94.197:44271/ using document root <none> and password file <none>
I20240502 14:14:46.328886 29094 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:46.329035 29094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:46.335443 29094 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20240502 14:14:46.338969 29108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:46.341035 29094 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240502 14:14:46.341226 29094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "1cb67158133e4648a6eceba32d4dbba3"
format_stamp: "Formatted at 2024-05-02 14:14:15 on dist-test-slave-3j2v"
I20240502 14:14:46.342176 29094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:46.367882 29094 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:46.368786 29094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:14:46.369065 29094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:46.370146 29094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:46.372056 29094 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:46.372135 29094 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:46.372221 29094 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:46.372264 29094 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:46.421082 29094 rpc_server.cc:307] RPC server started. Bound to: 127.19.94.197:38715
I20240502 14:14:46.421120 29220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.94.197:38715 every 8 connection(s)
I20240502 14:14:46.422703 29094 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 14:14:46.427389 19835 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_zeJNl/build/asan/bin/kudu as pid 29094
I20240502 14:14:46.428150 19835 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_zeJNl/build/asan/bin/kudu
/tmp/dist-test-task_zeJNl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.94.198:43659
--local_ip_for_outbound_sockets=127.19.94.198
--tserver_master_addrs=127.19.94.254:34367
--webserver_port=46553
--webserver_interface=127.19.94.198
--builtin_ntp_servers=127.19.94.212:40605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:14:46.443815 29221 heartbeater.cc:347] Connected to a master server at 127.19.94.254:34367
I20240502 14:14:46.444175 29221 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:46.445225 29221 heartbeater.cc:511] Master 127.19.94.254:34367 requested a full tablet report, sending...
I20240502 14:14:46.447110 26276 ts_manager.cc:194] Re-registered known tserver with Master: 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:14:46.448091 26276 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.94.197:37197
W20240502 14:14:46.826418 29225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:46.826771 29225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:14:46.826879 29225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:46.835844 29225 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:14:46.835990 29225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:14:46.836117 29225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.94.198
I20240502 14:14:46.847362 29225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.94.212:40605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.94.198:43659
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.19.94.198
--webserver_port=46553
--tserver_master_addrs=127.19.94.254:34367
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.94.198
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:14:46.848879 29225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:14:46.851251 29225 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:46.865936 29232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:46.866472 29233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:46.867058 29235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:46.867811 29225 server_base.cc:1034] running on GCE node
I20240502 14:14:46.868643 29225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:14:46.869925 29225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:14:46.871173 29225 hybrid_clock.cc:648] HybridClock initialized: now 1714659286871108 us; error 46 us; skew 500 ppm
I20240502 14:14:46.871634 29225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:46.874614 29225 webserver.cc:460] Webserver started at http://127.19.94.198:46553/ using document root <none> and password file <none>
I20240502 14:14:46.875577 29225 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:46.875721 29225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:46.882488 29225 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20240502 14:14:46.885766 29241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:46.887254 29225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240502 14:14:46.887418 29225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
format_stamp: "Formatted at 2024-05-02 14:14:16 on dist-test-slave-3j2v"
I20240502 14:14:46.888327 29225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:46.925526 29225 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:46.926445 29225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:14:46.926780 29225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:46.927863 29225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:46.929563 29225 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:46.929646 29225 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:46.929751 29225 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:46.929798 29225 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:46.979386 29225 rpc_server.cc:307] RPC server started. Bound to: 127.19.94.198:43659
I20240502 14:14:46.979498 29353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.94.198:43659 every 8 connection(s)
I20240502 14:14:46.981310 29225 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 14:14:46.986848 19835 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_zeJNl/build/asan/bin/kudu as pid 29225
I20240502 14:14:46.987591 19835 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_zeJNl/build/asan/bin/kudu
/tmp/dist-test-task_zeJNl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.94.199:41781
--local_ip_for_outbound_sockets=127.19.94.199
--tserver_master_addrs=127.19.94.254:34367
--webserver_port=44101
--webserver_interface=127.19.94.199
--builtin_ntp_servers=127.19.94.212:40605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:14:46.996876 29354 heartbeater.cc:347] Connected to a master server at 127.19.94.254:34367
I20240502 14:14:46.997198 29354 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:46.998013 29354 heartbeater.cc:511] Master 127.19.94.254:34367 requested a full tablet report, sending...
I20240502 14:14:46.999449 26276 ts_manager.cc:194] Re-registered known tserver with Master: c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:14:47.000378 26276 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.94.198:40973
W20240502 14:14:47.390992 29358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:47.391299 29358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:14:47.391389 29358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:47.400373 29358 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:14:47.400568 29358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:14:47.400743 29358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.94.199
I20240502 14:14:47.414173 29358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.94.212:40605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.94.199:41781
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.19.94.199
--webserver_port=44101
--tserver_master_addrs=127.19.94.254:34367
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.94.199
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:14:47.415973 29358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:14:47.417897 29358 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:47.433341 29366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:47.434125 29365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:47.434652 29368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:47.436066 29358 server_base.cc:1034] running on GCE node
I20240502 14:14:47.436794 29358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:14:47.437884 29358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:14:47.439103 29358 hybrid_clock.cc:648] HybridClock initialized: now 1714659287439049 us; error 49 us; skew 500 ppm
I20240502 14:14:47.439481 29358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:47.442132 29358 webserver.cc:460] Webserver started at http://127.19.94.199:44101/ using document root <none> and password file <none>
I20240502 14:14:47.443163 29358 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:47.443327 29358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:47.449688 29358 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20240502 14:14:47.451656 29221 heartbeater.cc:503] Master 127.19.94.254:34367 was elected leader, sending a full tablet report...
I20240502 14:14:47.453096 29375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:47.454594 29358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 14:14:47.454764 29358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "68a3c2b17ed94e06aba66ed4178132ab"
format_stamp: "Formatted at 2024-05-02 14:14:16 on dist-test-slave-3j2v"
I20240502 14:14:47.455777 29358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:47.484042 29358 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:47.484963 29358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:14:47.485245 29358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:47.486294 29358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:47.488899 29358 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:47.488992 29358 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:47.489136 29358 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:47.489190 29358 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:47.537066 29358 rpc_server.cc:307] RPC server started. Bound to: 127.19.94.199:41781
I20240502 14:14:47.537109 29487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.94.199:41781 every 8 connection(s)
I20240502 14:14:47.538676 29358 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 14:14:47.539196 19835 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_zeJNl/build/asan/bin/kudu as pid 29358
I20240502 14:14:47.540055 19835 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_zeJNl/build/asan/bin/kudu
/tmp/dist-test-task_zeJNl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.94.200:46097
--local_ip_for_outbound_sockets=127.19.94.200
--tserver_master_addrs=127.19.94.254:34367
--webserver_port=37805
--webserver_interface=127.19.94.200
--builtin_ntp_servers=127.19.94.212:40605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:14:47.563241 29488 heartbeater.cc:347] Connected to a master server at 127.19.94.254:34367
I20240502 14:14:47.563652 29488 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:47.564544 29488 heartbeater.cc:511] Master 127.19.94.254:34367 requested a full tablet report, sending...
I20240502 14:14:47.566499 26276 ts_manager.cc:194] Re-registered known tserver with Master: 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:14:47.567828 26276 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.94.199:47199
W20240502 14:14:47.941869 29492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:47.942231 29492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:14:47.942379 29492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:47.950968 29492 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:14:47.951141 29492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:14:47.951346 29492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.94.200
I20240502 14:14:47.962837 29492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.94.212:40605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.94.200:46097
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.19.94.200
--webserver_port=37805
--tserver_master_addrs=127.19.94.254:34367
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.94.200
--log_dir=/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:14:47.964231 29492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:14:47.966192 29492 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:47.981410 29497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:47.981714 29498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:47.983281 29492 server_base.cc:1034] running on GCE node
W20240502 14:14:47.983726 29501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:47.984555 29492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:14:47.985653 29492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:14:47.986910 29492 hybrid_clock.cc:648] HybridClock initialized: now 1714659287986847 us; error 53 us; skew 500 ppm
I20240502 14:14:47.987310 29492 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:47.990311 29492 webserver.cc:460] Webserver started at http://127.19.94.200:37805/ using document root <none> and password file <none>
I20240502 14:14:47.991158 29492 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:47.991312 29492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:47.998482 29492 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20240502 14:14:48.001613 29507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:48.003540 29492 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20240502 14:14:48.004199 29354 heartbeater.cc:503] Master 127.19.94.254:34367 was elected leader, sending a full tablet report...
I20240502 14:14:48.003739 29492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "4092902b3e674857a0147f1b4f2cdb86"
format_stamp: "Formatted at 2024-05-02 14:14:17 on dist-test-slave-3j2v"
I20240502 14:14:48.005093 29492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:48.029690 29492 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:48.030848 29492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:14:48.031208 29492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:48.032800 29492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:48.034689 29492 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:48.034770 29492 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:48.034879 29492 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:48.034925 29492 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:48.088845 29492 rpc_server.cc:307] RPC server started. Bound to: 127.19.94.200:46097
I20240502 14:14:48.088876 29619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.94.200:46097 every 8 connection(s)
I20240502 14:14:48.090394 29492 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 14:14:48.092330 19835 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_zeJNl/build/asan/bin/kudu as pid 29492
I20240502 14:14:48.100435 26276 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:14:48.111730 29620 heartbeater.cc:347] Connected to a master server at 127.19.94.254:34367
I20240502 14:14:48.112123 29620 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:48.113365 29620 heartbeater.cc:511] Master 127.19.94.254:34367 requested a full tablet report, sending...
I20240502 14:14:48.115146 26278 ts_manager.cc:194] Re-registered known tserver with Master: 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:14:48.117324 26537 tablet_service.cc:1467] Processing CreateTablet for tablet 55c42fbd424f4e7ca7f32806a891deb2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=16967cf57da940a2898db0011c297b55]), partition=
I20240502 14:14:48.117841 26537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55c42fbd424f4e7ca7f32806a891deb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:48.116328 26278 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.94.200:55611
I20240502 14:14:48.136648 29423 tablet_service.cc:1467] Processing CreateTablet for tablet 55c42fbd424f4e7ca7f32806a891deb2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=16967cf57da940a2898db0011c297b55]), partition=
I20240502 14:14:48.138279 29423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55c42fbd424f4e7ca7f32806a891deb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:48.146834 29289 tablet_service.cc:1467] Processing CreateTablet for tablet 55c42fbd424f4e7ca7f32806a891deb2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=16967cf57da940a2898db0011c297b55]), partition=
I20240502 14:14:48.148191 29289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55c42fbd424f4e7ca7f32806a891deb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:48.166450 29637 tablet_bootstrap.cc:492] T 55c42fbd424f4e7ca7f32806a891deb2 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:48.170276 29637 tablet_bootstrap.cc:654] T 55c42fbd424f4e7ca7f32806a891deb2 P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:48.171823 29638 tablet_bootstrap.cc:492] T 55c42fbd424f4e7ca7f32806a891deb2 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:14:48.175436 29639 tablet_bootstrap.cc:492] T 55c42fbd424f4e7ca7f32806a891deb2 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:14:48.176230 29638 tablet_bootstrap.cc:654] T 55c42fbd424f4e7ca7f32806a891deb2 P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:48.177953 29638 log.cc:824] T 55c42fbd424f4e7ca7f32806a891deb2 P 68a3c2b17ed94e06aba66ed4178132ab: Log is configured to *not* fsync() on all Append() calls
I20240502 14:14:48.180028 29639 tablet_bootstrap.cc:654] T 55c42fbd424f4e7ca7f32806a891deb2 P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:48.181214 29637 tablet_bootstrap.cc:492] T 55c42fbd424f4e7ca7f32806a891deb2 P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:48.181493 29637 ts_tablet_manager.cc:1367] T 55c42fbd424f4e7ca7f32806a891deb2 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.000s
I20240502 14:14:48.182597 29637 raft_consensus.cc:348] T 55c42fbd424f4e7ca7f32806a891deb2 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:48.183117 29637 raft_consensus.cc:374] T 55c42fbd424f4e7ca7f32806a891deb2 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:48.183284 29637 raft_consensus.cc:729] T 55c42fbd424f4e7ca7f32806a891deb2 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:48.184427 29639 log.cc:824] T 55c42fbd424f4e7ca7f32806a891deb2 P c1100e16b3b9495ca61d4f95e89b6a89: Log is configured to *not* fsync() on all Append() calls
I20240502 14:14:48.185418 29637 consensus_queue.cc:260] T 55c42fbd424f4e7ca7f32806a891deb2 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:48.186398 29637 ts_tablet_manager.cc:1398] T 55c42fbd424f4e7ca7f32806a891deb2 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:14:48.191306 29638 tablet_bootstrap.cc:492] T 55c42fbd424f4e7ca7f32806a891deb2 P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:14:48.191598 29638 ts_tablet_manager.cc:1367] T 55c42fbd424f4e7ca7f32806a891deb2 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.022s user 0.014s sys 0.000s
I20240502 14:14:48.191663 29639 tablet_bootstrap.cc:492] T 55c42fbd424f4e7ca7f32806a891deb2 P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:14:48.191960 29639 ts_tablet_manager.cc:1367] T 55c42fbd424f4e7ca7f32806a891deb2 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.005s
I20240502 14:14:48.200851 29639 raft_consensus.cc:348] T 55c42fbd424f4e7ca7f32806a891deb2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:48.201346 29639 raft_consensus.cc:374] T 55c42fbd424f4e7ca7f32806a891deb2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:48.201457 29639 raft_consensus.cc:729] T 55c42fbd424f4e7ca7f32806a891deb2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:14:48.202497 29639 consensus_queue.cc:260] T 55c42fbd424f4e7ca7f32806a891deb2 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:48.206115 29638 raft_consensus.cc:348] T 55c42fbd424f4e7ca7f32806a891deb2 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:48.206606 29638 raft_consensus.cc:374] T 55c42fbd424f4e7ca7f32806a891deb2 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:48.206707 29638 raft_consensus.cc:729] T 55c42fbd424f4e7ca7f32806a891deb2 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:14:48.207551 29638 consensus_queue.cc:260] T 55c42fbd424f4e7ca7f32806a891deb2 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:48.213104 29639 ts_tablet_manager.cc:1398] T 55c42fbd424f4e7ca7f32806a891deb2 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.021s user 0.014s sys 0.006s
W20240502 14:14:48.233958 29355 tablet.cc:2369] T 55c42fbd424f4e7ca7f32806a891deb2 P c1100e16b3b9495ca61d4f95e89b6a89: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:14:48.236364 29488 heartbeater.cc:503] Master 127.19.94.254:34367 was elected leader, sending a full tablet report...
I20240502 14:14:48.237329 29638 ts_tablet_manager.cc:1398] T 55c42fbd424f4e7ca7f32806a891deb2 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.046s user 0.018s sys 0.018s
I20240502 14:14:48.277184 29644 raft_consensus.cc:482] T 55c42fbd424f4e7ca7f32806a891deb2 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:48.277505 29644 raft_consensus.cc:504] T 55c42fbd424f4e7ca7f32806a891deb2 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:48.279400 29644 leader_election.cc:288] T 55c42fbd424f4e7ca7f32806a891deb2 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:14:48.284539 29643 raft_consensus.cc:482] T 55c42fbd424f4e7ca7f32806a891deb2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:48.284807 29643 raft_consensus.cc:504] T 55c42fbd424f4e7ca7f32806a891deb2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:48.286959 29643 leader_election.cc:288] T 55c42fbd424f4e7ca7f32806a891deb2 P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:14:48.290942 29489 tablet.cc:2369] T 55c42fbd424f4e7ca7f32806a891deb2 P 68a3c2b17ed94e06aba66ed4178132ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:14:48.325973 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55c42fbd424f4e7ca7f32806a891deb2" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:14:48.326555 29309 raft_consensus.cc:2454] T 55c42fbd424f4e7ca7f32806a891deb2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 0.
I20240502 14:14:48.327647 29378 leader_election.cc:302] T 55c42fbd424f4e7ca7f32806a891deb2 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:14:48.328178 29644 raft_consensus.cc:2781] T 55c42fbd424f4e7ca7f32806a891deb2 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:48.328364 29644 raft_consensus.cc:482] T 55c42fbd424f4e7ca7f32806a891deb2 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:48.328519 29644 raft_consensus.cc:3037] T 55c42fbd424f4e7ca7f32806a891deb2 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:48.331403 26556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55c42fbd424f4e7ca7f32806a891deb2" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:48.331688 26556 raft_consensus.cc:2454] T 55c42fbd424f4e7ca7f32806a891deb2 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 0.
I20240502 14:14:48.331614 29644 raft_consensus.cc:504] T 55c42fbd424f4e7ca7f32806a891deb2 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:48.332368 29644 leader_election.cc:288] T 55c42fbd424f4e7ca7f32806a891deb2 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 election: Requested vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:14:48.334371 26556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55c42fbd424f4e7ca7f32806a891deb2" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:14:48.334600 26556 raft_consensus.cc:3037] T 55c42fbd424f4e7ca7f32806a891deb2 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:48.334653 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55c42fbd424f4e7ca7f32806a891deb2" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
I20240502 14:14:48.334893 29309 raft_consensus.cc:3037] T 55c42fbd424f4e7ca7f32806a891deb2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:48.337682 26556 raft_consensus.cc:2454] T 55c42fbd424f4e7ca7f32806a891deb2 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 1.
I20240502 14:14:48.338008 29309 raft_consensus.cc:2454] T 55c42fbd424f4e7ca7f32806a891deb2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 1.
I20240502 14:14:48.338133 26557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55c42fbd424f4e7ca7f32806a891deb2" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:48.338325 29378 leader_election.cc:302] T 55c42fbd424f4e7ca7f32806a891deb2 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:48.338464 26557 raft_consensus.cc:2379] T 55c42fbd424f4e7ca7f32806a891deb2 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c1100e16b3b9495ca61d4f95e89b6a89 in current term 1: Already voted for candidate 68a3c2b17ed94e06aba66ed4178132ab in this term.
I20240502 14:14:48.338788 29644 raft_consensus.cc:2781] T 55c42fbd424f4e7ca7f32806a891deb2 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:48.339334 29644 raft_consensus.cc:686] T 55c42fbd424f4e7ca7f32806a891deb2 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 LEADER]: Becoming Leader. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Running, Role: LEADER
I20240502 14:14:48.340036 29644 consensus_queue.cc:237] T 55c42fbd424f4e7ca7f32806a891deb2 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:48.345343 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55c42fbd424f4e7ca7f32806a891deb2" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:14:48.346825 29245 leader_election.cc:302] T 55c42fbd424f4e7ca7f32806a891deb2 P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c1100e16b3b9495ca61d4f95e89b6a89; no voters: 68a3c2b17ed94e06aba66ed4178132ab, ba187e38a2754d60a12de9ac4856430a
I20240502 14:14:48.347654 29643 raft_consensus.cc:2726] T 55c42fbd424f4e7ca7f32806a891deb2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:14:48.347549 26278 catalog_manager.cc:5484] T 55c42fbd424f4e7ca7f32806a891deb2 P 68a3c2b17ed94e06aba66ed4178132ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199). New cstate: current_term: 1 leader_uuid: "68a3c2b17ed94e06aba66ed4178132ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:14:48.482266 26278 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:14:48.492432 26557 raft_consensus.cc:1261] T 55c42fbd424f4e7ca7f32806a891deb2 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:48.492655 29309 raft_consensus.cc:1261] T 55c42fbd424f4e7ca7f32806a891deb2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:48.493240 29652 consensus_queue.cc:1035] T 55c42fbd424f4e7ca7f32806a891deb2 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:48.493610 29644 consensus_queue.cc:1035] T 55c42fbd424f4e7ca7f32806a891deb2 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:48.518816 26276 catalog_manager.cc:5576] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659) does not have the latest schema for tablet 55c42fbd424f4e7ca7f32806a891deb2 (table rebalancer_dynamic_table [id=16967cf57da940a2898db0011c297b55]). Expected version 1 got 0
I20240502 14:14:48.526111 29655 mvcc.cc:205] Tried to move back new op lower bound from 7023244445655650304 to 7023244445048930304. Current Snapshot: MvccSnapshot[applied={T|T < 7023244445655650304}]
I20240502 14:14:48.529735 26278 catalog_manager.cc:5576] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025) does not have the latest schema for tablet 55c42fbd424f4e7ca7f32806a891deb2 (table rebalancer_dynamic_table [id=16967cf57da940a2898db0011c297b55]). Expected version 1 got 0
I20240502 14:14:48.531832 29660 tablet.cc:1722] T 55c42fbd424f4e7ca7f32806a891deb2 P ba187e38a2754d60a12de9ac4856430a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:48.534579 29660 tablet.cc:1620] T 55c42fbd424f4e7ca7f32806a891deb2 P ba187e38a2754d60a12de9ac4856430a: MemRowSet was empty: no flush needed.
I20240502 14:14:48.535704 26276 catalog_manager.cc:5576] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025) does not have the latest schema for tablet 55c42fbd424f4e7ca7f32806a891deb2 (table rebalancer_dynamic_table [id=16967cf57da940a2898db0011c297b55]). Expected version 1 got 0
I20240502 14:14:48.541882 26276 catalog_manager.cc:5576] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659) does not have the latest schema for tablet 55c42fbd424f4e7ca7f32806a891deb2 (table rebalancer_dynamic_table [id=16967cf57da940a2898db0011c297b55]). Expected version 1 got 0
I20240502 14:14:48.549968 26283 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=16967cf57da940a2898db0011c297b55] alter complete (version 1)
I20240502 14:14:48.557906 29662 tablet.cc:1722] T 55c42fbd424f4e7ca7f32806a891deb2 P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:48.560348 29662 tablet.cc:1620] T 55c42fbd424f4e7ca7f32806a891deb2 P c1100e16b3b9495ca61d4f95e89b6a89: MemRowSet was empty: no flush needed.
I20240502 14:14:48.562156 29661 tablet.cc:1722] T 55c42fbd424f4e7ca7f32806a891deb2 P 68a3c2b17ed94e06aba66ed4178132ab: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:48.564013 29661 tablet.cc:1620] T 55c42fbd424f4e7ca7f32806a891deb2 P 68a3c2b17ed94e06aba66ed4178132ab: MemRowSet was empty: no flush needed.
I20240502 14:14:48.595316 26278 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_extra_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:48.597956 26283 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240502 14:14:48.598222 26283 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240502 14:14:48.604125 26283 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet bbe54b53e86a48fbb1ba02074fb7afe3
I20240502 14:14:48.605660 26799 tablet_service.cc:1514] Processing DeleteTablet for tablet bbe54b53e86a48fbb1ba02074fb7afe3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:48 UTC) from {username='slave'} at 127.0.0.1:56010
I20240502 14:14:48.607559 26669 tablet_service.cc:1514] Processing DeleteTablet for tablet bbe54b53e86a48fbb1ba02074fb7afe3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:48 UTC) from {username='slave'} at 127.0.0.1:34150
I20240502 14:14:48.608577 26283 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet 63c1903111784cf38265e2eeb4faff2e
I20240502 14:14:48.608994 26407 tablet_service.cc:1514] Processing DeleteTablet for tablet bbe54b53e86a48fbb1ba02074fb7afe3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:48 UTC) from {username='slave'} at 127.0.0.1:37422
I20240502 14:14:48.609424 26537 tablet_service.cc:1514] Processing DeleteTablet for tablet 63c1903111784cf38265e2eeb4faff2e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:48 UTC) from {username='slave'} at 127.0.0.1:51696
I20240502 14:14:48.609516 26283 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet 52061de1d76a420db5f3eb0aad70bf3a
I20240502 14:14:48.609918 26406 tablet_service.cc:1514] Processing DeleteTablet for tablet 63c1903111784cf38265e2eeb4faff2e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:48 UTC) from {username='slave'} at 127.0.0.1:37422
I20240502 14:14:48.610422 26538 tablet_service.cc:1514] Processing DeleteTablet for tablet 52061de1d76a420db5f3eb0aad70bf3a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:48 UTC) from {username='slave'} at 127.0.0.1:51696
I20240502 14:14:48.611297 26797 tablet_service.cc:1467] Processing CreateTablet for tablet 06ae914a36b244f982cd2ace31769a79 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=7727fea7ae174c949e336e765c50e53d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:48.611562 26798 tablet_service.cc:1514] Processing DeleteTablet for tablet 52061de1d76a420db5f3eb0aad70bf3a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:48 UTC) from {username='slave'} at 127.0.0.1:56010
I20240502 14:14:48.611851 26797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06ae914a36b244f982cd2ace31769a79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:48.611882 26406 tablet_service.cc:1514] Processing DeleteTablet for tablet 52061de1d76a420db5f3eb0aad70bf3a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:48 UTC) from {username='slave'} at 127.0.0.1:37422
I20240502 14:14:48.614269 29668 tablet_replica.cc:330] T bbe54b53e86a48fbb1ba02074fb7afe3 P 89af7f423cf44717a08d6b142f0ed49d: stopping tablet replica
I20240502 14:14:48.617405 29666 tablet_replica.cc:330] T bbe54b53e86a48fbb1ba02074fb7afe3 P e4cfb8fc5a474177af6848a6efa13a27: stopping tablet replica
I20240502 14:14:48.618072 29668 raft_consensus.cc:2229] T bbe54b53e86a48fbb1ba02074fb7afe3 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:48.616138 26668 tablet_service.cc:1514] Processing DeleteTablet for tablet 63c1903111784cf38265e2eeb4faff2e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:48 UTC) from {username='slave'} at 127.0.0.1:34150
I20240502 14:14:48.615155 29667 tablet_replica.cc:330] T bbe54b53e86a48fbb1ba02074fb7afe3 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: stopping tablet replica
I20240502 14:14:48.619117 29667 raft_consensus.cc:2229] T bbe54b53e86a48fbb1ba02074fb7afe3 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:48.619663 29666 raft_consensus.cc:2229] T bbe54b53e86a48fbb1ba02074fb7afe3 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:48.619660 26407 tablet_service.cc:1467] Processing CreateTablet for tablet 06ae914a36b244f982cd2ace31769a79 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=7727fea7ae174c949e336e765c50e53d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:48.620551 26407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06ae914a36b244f982cd2ace31769a79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:48.620774 29668 raft_consensus.cc:2258] T bbe54b53e86a48fbb1ba02074fb7afe3 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:48.621372 29667 raft_consensus.cc:2258] T bbe54b53e86a48fbb1ba02074fb7afe3 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:48.623224 29667 ts_tablet_manager.cc:1875] T bbe54b53e86a48fbb1ba02074fb7afe3 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:48.624035 29666 raft_consensus.cc:2258] T bbe54b53e86a48fbb1ba02074fb7afe3 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:48.630077 29289 tablet_service.cc:1467] Processing CreateTablet for tablet e93acdb9427943938f4c12ae2bf2c594 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=7727fea7ae174c949e336e765c50e53d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:48.630805 29289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e93acdb9427943938f4c12ae2bf2c594. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:48.635531 29423 tablet_service.cc:1467] Processing CreateTablet for tablet e93acdb9427943938f4c12ae2bf2c594 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=7727fea7ae174c949e336e765c50e53d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:48.636288 29423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e93acdb9427943938f4c12ae2bf2c594. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:48.637573 26406 tablet_service.cc:1467] Processing CreateTablet for tablet 177a34c96aa84cb4aa50bbd79cab4144 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=7727fea7ae174c949e336e765c50e53d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:48.638039 26406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 177a34c96aa84cb4aa50bbd79cab4144. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:48.638715 29421 tablet_service.cc:1467] Processing CreateTablet for tablet 177a34c96aa84cb4aa50bbd79cab4144 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=7727fea7ae174c949e336e765c50e53d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:48.639190 29421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 177a34c96aa84cb4aa50bbd79cab4144. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:48.646445 29639 tablet_bootstrap.cc:492] T e93acdb9427943938f4c12ae2bf2c594 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:14:48.648238 29639 tablet_bootstrap.cc:654] T e93acdb9427943938f4c12ae2bf2c594 P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:48.655095 29669 tablet_replica.cc:330] T 63c1903111784cf38265e2eeb4faff2e P ba187e38a2754d60a12de9ac4856430a: stopping tablet replica
I20240502 14:14:48.657769 29669 raft_consensus.cc:2229] T 63c1903111784cf38265e2eeb4faff2e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:48.658756 26283 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:14:48.659735 29669 raft_consensus.cc:2258] T 63c1903111784cf38265e2eeb4faff2e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:48.665704 29669 ts_tablet_manager.cc:1875] T 63c1903111784cf38265e2eeb4faff2e P ba187e38a2754d60a12de9ac4856430a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:48.667145 29639 tablet_bootstrap.cc:492] T e93acdb9427943938f4c12ae2bf2c594 P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:14:48.667338 29639 ts_tablet_manager.cc:1367] T e93acdb9427943938f4c12ae2bf2c594 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.002s
I20240502 14:14:48.668934 29639 raft_consensus.cc:348] T e93acdb9427943938f4c12ae2bf2c594 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:48.669199 29639 raft_consensus.cc:374] T e93acdb9427943938f4c12ae2bf2c594 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:48.669301 29639 raft_consensus.cc:729] T e93acdb9427943938f4c12ae2bf2c594 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:14:48.674989 29639 consensus_queue.cc:260] T e93acdb9427943938f4c12ae2bf2c594 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:48.678133 29661 tablet.cc:1722] T 55c42fbd424f4e7ca7f32806a891deb2 P 68a3c2b17ed94e06aba66ed4178132ab: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:48.678529 29639 ts_tablet_manager.cc:1398] T e93acdb9427943938f4c12ae2bf2c594 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.011s user 0.002s sys 0.001s
I20240502 14:14:48.678735 29661 tablet.cc:1620] T 55c42fbd424f4e7ca7f32806a891deb2 P 68a3c2b17ed94e06aba66ed4178132ab: MemRowSet was empty: no flush needed.
I20240502 14:14:48.678861 29643 raft_consensus.cc:482] T e93acdb9427943938f4c12ae2bf2c594 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:48.679015 29643 raft_consensus.cc:504] T e93acdb9427943938f4c12ae2bf2c594 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:48.679876 29643 leader_election.cc:288] T e93acdb9427943938f4c12ae2bf2c594 P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:14:48.688230 29672 tablet_bootstrap.cc:492] T 06ae914a36b244f982cd2ace31769a79 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:48.688645 29673 tablet_bootstrap.cc:492] T 06ae914a36b244f982cd2ace31769a79 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:48.692078 29672 tablet_bootstrap.cc:654] T 06ae914a36b244f982cd2ace31769a79 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:48.692131 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93acdb9427943938f4c12ae2bf2c594" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:14:48.694643 29673 tablet_bootstrap.cc:654] T 06ae914a36b244f982cd2ace31769a79 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:48.697624 29662 tablet.cc:1722] T 55c42fbd424f4e7ca7f32806a891deb2 P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:48.698352 29662 tablet.cc:1620] T 55c42fbd424f4e7ca7f32806a891deb2 P c1100e16b3b9495ca61d4f95e89b6a89: MemRowSet was empty: no flush needed.
W20240502 14:14:48.707271 29245 leader_election.cc:341] T e93acdb9427943938f4c12ae2bf2c594 P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:48.713649 29660 tablet.cc:1722] T 55c42fbd424f4e7ca7f32806a891deb2 P ba187e38a2754d60a12de9ac4856430a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:48.714212 29660 tablet.cc:1620] T 55c42fbd424f4e7ca7f32806a891deb2 P ba187e38a2754d60a12de9ac4856430a: MemRowSet was empty: no flush needed.
I20240502 14:14:48.720942 29666 ts_tablet_manager.cc:1875] T bbe54b53e86a48fbb1ba02074fb7afe3 P e4cfb8fc5a474177af6848a6efa13a27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:48.722232 29638 tablet_bootstrap.cc:492] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:14:48.727146 29638 tablet_bootstrap.cc:654] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:48.735042 26278 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=16967cf57da940a2898db0011c297b55] alter complete (version 2)
I20240502 14:14:48.711763 29555 tablet_service.cc:1467] Processing CreateTablet for tablet 06ae914a36b244f982cd2ace31769a79 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=7727fea7ae174c949e336e765c50e53d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:48.742581 29555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06ae914a36b244f982cd2ace31769a79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:48.631984 29668 ts_tablet_manager.cc:1875] T bbe54b53e86a48fbb1ba02074fb7afe3 P 89af7f423cf44717a08d6b142f0ed49d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:48.769979 29673 tablet_bootstrap.cc:492] T 06ae914a36b244f982cd2ace31769a79 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:48.770273 29673 ts_tablet_manager.cc:1367] T 06ae914a36b244f982cd2ace31769a79 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.082s user 0.004s sys 0.028s
I20240502 14:14:48.771221 29673 raft_consensus.cc:348] T 06ae914a36b244f982cd2ace31769a79 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:48.771559 29673 raft_consensus.cc:374] T 06ae914a36b244f982cd2ace31769a79 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:48.771697 29673 raft_consensus.cc:729] T 06ae914a36b244f982cd2ace31769a79 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:48.772045 29673 consensus_queue.cc:260] T 06ae914a36b244f982cd2ace31769a79 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:48.774634 29673 ts_tablet_manager.cc:1398] T 06ae914a36b244f982cd2ace31769a79 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240502 14:14:48.775014 29673 tablet_bootstrap.cc:492] T 177a34c96aa84cb4aa50bbd79cab4144 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:48.777344 29669 ts_tablet_manager.cc:1888] T 63c1903111784cf38265e2eeb4faff2e P ba187e38a2754d60a12de9ac4856430a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:48.777541 29669 log.cc:1196] T 63c1903111784cf38265e2eeb4faff2e P ba187e38a2754d60a12de9ac4856430a: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/63c1903111784cf38265e2eeb4faff2e
I20240502 14:14:48.782187 29669 ts_tablet_manager.cc:1909] T 63c1903111784cf38265e2eeb4faff2e P ba187e38a2754d60a12de9ac4856430a: Deleting consensus metadata
I20240502 14:14:48.783727 29673 tablet_bootstrap.cc:654] T 177a34c96aa84cb4aa50bbd79cab4144 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:48.786581 26271 catalog_manager.cc:4830] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): tablet 63c1903111784cf38265e2eeb4faff2e (table rebalance_test_table_0 [id=911ed82f4a3a4ad4938e4b1c8608e51b]) successfully deleted
I20240502 14:14:48.790338 29669 tablet_replica.cc:330] T 52061de1d76a420db5f3eb0aad70bf3a P ba187e38a2754d60a12de9ac4856430a: stopping tablet replica
I20240502 14:14:48.790793 29669 raft_consensus.cc:2229] T 52061de1d76a420db5f3eb0aad70bf3a P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:48.791349 29669 raft_consensus.cc:2258] T 52061de1d76a420db5f3eb0aad70bf3a P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:48.794618 29638 tablet_bootstrap.cc:492] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:14:48.794860 29638 ts_tablet_manager.cc:1367] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.073s user 0.003s sys 0.006s
I20240502 14:14:48.796439 29638 raft_consensus.cc:348] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:48.796787 29638 raft_consensus.cc:374] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:48.796902 29638 raft_consensus.cc:729] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:14:48.797392 29638 consensus_queue.cc:260] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:48.798378 29669 ts_tablet_manager.cc:1875] T 52061de1d76a420db5f3eb0aad70bf3a P ba187e38a2754d60a12de9ac4856430a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:48.805905 29638 ts_tablet_manager.cc:1398] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.011s user 0.000s sys 0.003s
I20240502 14:14:48.808135 29638 tablet_bootstrap.cc:492] T 177a34c96aa84cb4aa50bbd79cab4144 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:14:48.819576 29673 tablet_bootstrap.cc:492] T 177a34c96aa84cb4aa50bbd79cab4144 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:48.820562 29673 ts_tablet_manager.cc:1367] T 177a34c96aa84cb4aa50bbd79cab4144 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.046s user 0.006s sys 0.011s
I20240502 14:14:48.822132 29673 raft_consensus.cc:348] T 177a34c96aa84cb4aa50bbd79cab4144 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:48.823571 29673 raft_consensus.cc:374] T 177a34c96aa84cb4aa50bbd79cab4144 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:48.823882 29673 raft_consensus.cc:729] T 177a34c96aa84cb4aa50bbd79cab4144 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:48.824467 29673 consensus_queue.cc:260] T 177a34c96aa84cb4aa50bbd79cab4144 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:48.827345 29669 ts_tablet_manager.cc:1888] T 52061de1d76a420db5f3eb0aad70bf3a P ba187e38a2754d60a12de9ac4856430a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:48.827533 29669 log.cc:1196] T 52061de1d76a420db5f3eb0aad70bf3a P ba187e38a2754d60a12de9ac4856430a: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/52061de1d76a420db5f3eb0aad70bf3a
I20240502 14:14:48.828791 29669 ts_tablet_manager.cc:1909] T 52061de1d76a420db5f3eb0aad70bf3a P ba187e38a2754d60a12de9ac4856430a: Deleting consensus metadata
I20240502 14:14:48.831506 29682 tablet_bootstrap.cc:492] T 06ae914a36b244f982cd2ace31769a79 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:14:48.832597 29673 ts_tablet_manager.cc:1398] T 177a34c96aa84cb4aa50bbd79cab4144 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240502 14:14:48.835081 26271 catalog_manager.cc:4830] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): tablet 52061de1d76a420db5f3eb0aad70bf3a (table rebalance_test_table_0 [id=911ed82f4a3a4ad4938e4b1c8608e51b]) successfully deleted
W20240502 14:14:48.838887 26754 consensus_peers.cc:485] T 52061de1d76a420db5f3eb0aad70bf3a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): Couldn't send request to peer ba187e38a2754d60a12de9ac4856430a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 52061de1d76a420db5f3eb0aad70bf3a. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:48.839390 26276 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:14:48.839864 29682 tablet_bootstrap.cc:654] T 06ae914a36b244f982cd2ace31769a79 P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:48.841562 29682 log.cc:824] T 06ae914a36b244f982cd2ace31769a79 P 4092902b3e674857a0147f1b4f2cdb86: Log is configured to *not* fsync() on all Append() calls
I20240502 14:14:48.847321 29672 tablet_bootstrap.cc:492] T 06ae914a36b244f982cd2ace31769a79 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:48.847609 29672 ts_tablet_manager.cc:1367] T 06ae914a36b244f982cd2ace31769a79 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.160s user 0.009s sys 0.043s
I20240502 14:14:48.849164 29672 raft_consensus.cc:348] T 06ae914a36b244f982cd2ace31769a79 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:48.849491 29672 raft_consensus.cc:374] T 06ae914a36b244f982cd2ace31769a79 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:48.849630 29672 raft_consensus.cc:729] T 06ae914a36b244f982cd2ace31769a79 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:48.849948 29672 consensus_queue.cc:260] T 06ae914a36b244f982cd2ace31769a79 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:48.850668 29672 ts_tablet_manager.cc:1398] T 06ae914a36b244f982cd2ace31769a79 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20240502 14:14:48.853842 29244 leader_election.cc:341] T e93acdb9427943938f4c12ae2bf2c594 P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): Not found: Tablet not found: e93acdb9427943938f4c12ae2bf2c594
I20240502 14:14:48.854094 29244 leader_election.cc:302] T e93acdb9427943938f4c12ae2bf2c594 P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c1100e16b3b9495ca61d4f95e89b6a89; no voters: 1cb67158133e4648a6eceba32d4dbba3, 68a3c2b17ed94e06aba66ed4178132ab
I20240502 14:14:48.854547 29643 raft_consensus.cc:2726] T e93acdb9427943938f4c12ae2bf2c594 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:14:48.862502 29638 tablet_bootstrap.cc:654] T 177a34c96aa84cb4aa50bbd79cab4144 P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:48.864243 29661 tablet.cc:1722] T 55c42fbd424f4e7ca7f32806a891deb2 P 68a3c2b17ed94e06aba66ed4178132ab: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:48.867440 29668 ts_tablet_manager.cc:1888] T bbe54b53e86a48fbb1ba02074fb7afe3 P 89af7f423cf44717a08d6b142f0ed49d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:48.867617 29668 log.cc:1196] T bbe54b53e86a48fbb1ba02074fb7afe3 P 89af7f423cf44717a08d6b142f0ed49d: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/bbe54b53e86a48fbb1ba02074fb7afe3
I20240502 14:14:48.868103 29668 ts_tablet_manager.cc:1909] T bbe54b53e86a48fbb1ba02074fb7afe3 P 89af7f423cf44717a08d6b142f0ed49d: Deleting consensus metadata
I20240502 14:14:48.868589 29660 tablet.cc:1722] T 55c42fbd424f4e7ca7f32806a891deb2 P ba187e38a2754d60a12de9ac4856430a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:48.876504 26276 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table_new_name [id=16967cf57da940a2898db0011c297b55] alter complete (version 3)
I20240502 14:14:48.882081 29662 tablet.cc:1722] T 55c42fbd424f4e7ca7f32806a891deb2 P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:48.882668 29667 ts_tablet_manager.cc:1888] T bbe54b53e86a48fbb1ba02074fb7afe3 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:48.883816 29666 ts_tablet_manager.cc:1888] T bbe54b53e86a48fbb1ba02074fb7afe3 P e4cfb8fc5a474177af6848a6efa13a27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:48.884127 29666 log.cc:1196] T bbe54b53e86a48fbb1ba02074fb7afe3 P e4cfb8fc5a474177af6848a6efa13a27: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/wals/bbe54b53e86a48fbb1ba02074fb7afe3
I20240502 14:14:48.884747 29666 ts_tablet_manager.cc:1909] T bbe54b53e86a48fbb1ba02074fb7afe3 P e4cfb8fc5a474177af6848a6efa13a27: Deleting consensus metadata
I20240502 14:14:48.885376 29638 tablet_bootstrap.cc:492] T 177a34c96aa84cb4aa50bbd79cab4144 P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:14:48.885578 29638 ts_tablet_manager.cc:1367] T 177a34c96aa84cb4aa50bbd79cab4144 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.078s user 0.004s sys 0.000s
I20240502 14:14:48.886413 29638 raft_consensus.cc:348] T 177a34c96aa84cb4aa50bbd79cab4144 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:48.886641 29666 tablet_replica.cc:330] T 52061de1d76a420db5f3eb0aad70bf3a P e4cfb8fc5a474177af6848a6efa13a27: stopping tablet replica
I20240502 14:14:48.886889 29666 raft_consensus.cc:2229] T 52061de1d76a420db5f3eb0aad70bf3a P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:48.887388 29666 raft_consensus.cc:2258] T 52061de1d76a420db5f3eb0aad70bf3a P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:48.888164 29666 ts_tablet_manager.cc:1875] T 52061de1d76a420db5f3eb0aad70bf3a P e4cfb8fc5a474177af6848a6efa13a27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:48.891746 26272 catalog_manager.cc:4830] TS e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): tablet bbe54b53e86a48fbb1ba02074fb7afe3 (table rebalance_test_table_0 [id=911ed82f4a3a4ad4938e4b1c8608e51b]) successfully deleted
I20240502 14:14:48.893525 29155 tablet_service.cc:1467] Processing CreateTablet for tablet 177a34c96aa84cb4aa50bbd79cab4144 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=7727fea7ae174c949e336e765c50e53d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:48.894987 29155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 177a34c96aa84cb4aa50bbd79cab4144. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:48.895267 29666 ts_tablet_manager.cc:1888] T 52061de1d76a420db5f3eb0aad70bf3a P e4cfb8fc5a474177af6848a6efa13a27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:48.895438 29666 log.cc:1196] T 52061de1d76a420db5f3eb0aad70bf3a P e4cfb8fc5a474177af6848a6efa13a27: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/wals/52061de1d76a420db5f3eb0aad70bf3a
I20240502 14:14:48.895493 29638 raft_consensus.cc:374] T 177a34c96aa84cb4aa50bbd79cab4144 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:48.895622 29638 raft_consensus.cc:729] T 177a34c96aa84cb4aa50bbd79cab4144 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:14:48.895933 29666 ts_tablet_manager.cc:1909] T 52061de1d76a420db5f3eb0aad70bf3a P e4cfb8fc5a474177af6848a6efa13a27: Deleting consensus metadata
I20240502 14:14:48.895972 29638 consensus_queue.cc:260] T 177a34c96aa84cb4aa50bbd79cab4144 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:48.896566 29667 log.cc:1196] T bbe54b53e86a48fbb1ba02074fb7afe3 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/wals/bbe54b53e86a48fbb1ba02074fb7afe3
I20240502 14:14:48.896519 29156 tablet_service.cc:1467] Processing CreateTablet for tablet e93acdb9427943938f4c12ae2bf2c594 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=7727fea7ae174c949e336e765c50e53d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:48.897238 29667 ts_tablet_manager.cc:1909] T bbe54b53e86a48fbb1ba02074fb7afe3 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting consensus metadata
I20240502 14:14:48.897730 29156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e93acdb9427943938f4c12ae2bf2c594. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:48.898651 29668 tablet_replica.cc:330] T 63c1903111784cf38265e2eeb4faff2e P 89af7f423cf44717a08d6b142f0ed49d: stopping tablet replica
I20240502 14:14:48.898914 29668 raft_consensus.cc:2229] T 63c1903111784cf38265e2eeb4faff2e P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:48.898970 26272 catalog_manager.cc:4830] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): tablet bbe54b53e86a48fbb1ba02074fb7afe3 (table rebalance_test_table_0 [id=911ed82f4a3a4ad4938e4b1c8608e51b]) successfully deleted
I20240502 14:14:48.899221 29668 raft_consensus.cc:2258] T 63c1903111784cf38265e2eeb4faff2e P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:48.899360 29667 tablet_replica.cc:330] T 63c1903111784cf38265e2eeb4faff2e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: stopping tablet replica
I20240502 14:14:48.899626 29667 raft_consensus.cc:2229] T 63c1903111784cf38265e2eeb4faff2e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:48.900085 29668 ts_tablet_manager.cc:1875] T 63c1903111784cf38265e2eeb4faff2e P 89af7f423cf44717a08d6b142f0ed49d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:48.900115 29667 raft_consensus.cc:2258] T 63c1903111784cf38265e2eeb4faff2e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:48.900926 29667 ts_tablet_manager.cc:1875] T 63c1903111784cf38265e2eeb4faff2e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:48.904695 29638 ts_tablet_manager.cc:1398] T 177a34c96aa84cb4aa50bbd79cab4144 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.019s user 0.002s sys 0.001s
I20240502 14:14:48.905699 26271 catalog_manager.cc:4830] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): tablet bbe54b53e86a48fbb1ba02074fb7afe3 (table rebalance_test_table_0 [id=911ed82f4a3a4ad4938e4b1c8608e51b]) successfully deleted
I20240502 14:14:48.908970 29659 raft_consensus.cc:482] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:48.909166 29659 raft_consensus.cc:504] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:48.910102 29659 leader_election.cc:288] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:14:48.914167 29667 ts_tablet_manager.cc:1888] T 63c1903111784cf38265e2eeb4faff2e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:48.914412 29667 log.cc:1196] T 63c1903111784cf38265e2eeb4faff2e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/wals/63c1903111784cf38265e2eeb4faff2e
I20240502 14:14:48.914952 29667 ts_tablet_manager.cc:1909] T 63c1903111784cf38265e2eeb4faff2e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting consensus metadata
I20240502 14:14:48.916182 26272 catalog_manager.cc:4830] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): tablet 63c1903111784cf38265e2eeb4faff2e (table rebalance_test_table_0 [id=911ed82f4a3a4ad4938e4b1c8608e51b]) successfully deleted
I20240502 14:14:48.918170 26272 catalog_manager.cc:4830] TS e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): tablet 52061de1d76a420db5f3eb0aad70bf3a (table rebalance_test_table_0 [id=911ed82f4a3a4ad4938e4b1c8608e51b]) successfully deleted
I20240502 14:14:48.926935 29668 ts_tablet_manager.cc:1888] T 63c1903111784cf38265e2eeb4faff2e P 89af7f423cf44717a08d6b142f0ed49d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:48.927222 29668 log.cc:1196] T 63c1903111784cf38265e2eeb4faff2e P 89af7f423cf44717a08d6b142f0ed49d: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/63c1903111784cf38265e2eeb4faff2e
I20240502 14:14:48.927773 29668 ts_tablet_manager.cc:1909] T 63c1903111784cf38265e2eeb4faff2e P 89af7f423cf44717a08d6b142f0ed49d: Deleting consensus metadata
I20240502 14:14:48.928925 29668 tablet_replica.cc:330] T 52061de1d76a420db5f3eb0aad70bf3a P 89af7f423cf44717a08d6b142f0ed49d: stopping tablet replica
I20240502 14:14:48.929102 26271 catalog_manager.cc:4830] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): tablet 63c1903111784cf38265e2eeb4faff2e (table rebalance_test_table_0 [id=911ed82f4a3a4ad4938e4b1c8608e51b]) successfully deleted
I20240502 14:14:48.929293 29668 raft_consensus.cc:2229] T 52061de1d76a420db5f3eb0aad70bf3a P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:48.929631 29668 raft_consensus.cc:2258] T 52061de1d76a420db5f3eb0aad70bf3a P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:48.930428 29668 ts_tablet_manager.cc:1875] T 52061de1d76a420db5f3eb0aad70bf3a P 89af7f423cf44717a08d6b142f0ed49d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:48.938277 29668 ts_tablet_manager.cc:1888] T 52061de1d76a420db5f3eb0aad70bf3a P 89af7f423cf44717a08d6b142f0ed49d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:48.938586 29668 log.cc:1196] T 52061de1d76a420db5f3eb0aad70bf3a P 89af7f423cf44717a08d6b142f0ed49d: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/52061de1d76a420db5f3eb0aad70bf3a
I20240502 14:14:48.939136 29668 ts_tablet_manager.cc:1909] T 52061de1d76a420db5f3eb0aad70bf3a P 89af7f423cf44717a08d6b142f0ed49d: Deleting consensus metadata
I20240502 14:14:48.940475 26271 catalog_manager.cc:4830] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): tablet 52061de1d76a420db5f3eb0aad70bf3a (table rebalance_test_table_0 [id=911ed82f4a3a4ad4938e4b1c8608e51b]) successfully deleted
I20240502 14:14:48.943310 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93acdb9427943938f4c12ae2bf2c594" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:14:48.943560 29309 raft_consensus.cc:2454] T e93acdb9427943938f4c12ae2bf2c594 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 0.
I20240502 14:14:48.945019 29378 leader_election.cc:302] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:14:48.945492 29659 raft_consensus.cc:2781] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:48.945637 29659 raft_consensus.cc:482] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:48.945748 29659 raft_consensus.cc:3037] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:48.948666 29659 raft_consensus.cc:504] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:48.949436 29659 leader_election.cc:288] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 election: Requested vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:14:48.950024 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93acdb9427943938f4c12ae2bf2c594" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
I20240502 14:14:48.950250 29309 raft_consensus.cc:3037] T e93acdb9427943938f4c12ae2bf2c594 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:48.953426 29309 raft_consensus.cc:2454] T e93acdb9427943938f4c12ae2bf2c594 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 1.
I20240502 14:14:48.953949 29378 leader_election.cc:302] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:14:48.954299 29659 raft_consensus.cc:2781] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:48.954514 29659 raft_consensus.cc:686] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 LEADER]: Becoming Leader. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Running, Role: LEADER
I20240502 14:14:48.954885 29659 consensus_queue.cc:237] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:48.958181 26283 catalog_manager.cc:5484] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199). New cstate: current_term: 1 leader_uuid: "68a3c2b17ed94e06aba66ed4178132ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
W20240502 14:14:48.961937 29378 leader_election.cc:341] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): Not found: Tablet not found: e93acdb9427943938f4c12ae2bf2c594
W20240502 14:14:48.962448 29378 leader_election.cc:341] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): Not found: Tablet not found: e93acdb9427943938f4c12ae2bf2c594
I20240502 14:14:48.962713 29682 tablet_bootstrap.cc:492] T 06ae914a36b244f982cd2ace31769a79 P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:14:48.963013 29682 ts_tablet_manager.cc:1367] T 06ae914a36b244f982cd2ace31769a79 P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.132s user 0.021s sys 0.022s
I20240502 14:14:48.967164 29693 tablet_bootstrap.cc:492] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:14:48.970992 29682 raft_consensus.cc:348] T 06ae914a36b244f982cd2ace31769a79 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:48.971544 29682 raft_consensus.cc:374] T 06ae914a36b244f982cd2ace31769a79 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:48.971689 29682 raft_consensus.cc:729] T 06ae914a36b244f982cd2ace31769a79 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:14:48.972507 29682 consensus_queue.cc:260] T 06ae914a36b244f982cd2ace31769a79 P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:48.973649 29693 tablet_bootstrap.cc:654] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:48.973721 29620 heartbeater.cc:503] Master 127.19.94.254:34367 was elected leader, sending a full tablet report...
I20240502 14:14:48.974181 29682 ts_tablet_manager.cc:1398] T 06ae914a36b244f982cd2ace31769a79 P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20240502 14:14:48.975054 29693 log.cc:824] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3: Log is configured to *not* fsync() on all Append() calls
I20240502 14:14:48.978139 29693 tablet_bootstrap.cc:492] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:14:48.978467 29693 ts_tablet_manager.cc:1367] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20240502 14:14:48.985918 29694 raft_consensus.cc:482] T 06ae914a36b244f982cd2ace31769a79 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:48.986200 29694 raft_consensus.cc:504] T 06ae914a36b244f982cd2ace31769a79 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:48.987785 29693 raft_consensus.cc:348] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:48.988227 29693 raft_consensus.cc:374] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:48.988323 29694 leader_election.cc:288] T 06ae914a36b244f982cd2ace31769a79 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:14:48.988353 29693 raft_consensus.cc:729] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:14:48.989295 29693 consensus_queue.cc:260] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:48.996564 29693 ts_tablet_manager.cc:1398] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.018s user 0.013s sys 0.004s
I20240502 14:14:49.002373 26277 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:14:49.002691 26277 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:14:49.007246 26277 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet 55c42fbd424f4e7ca7f32806a891deb2
I20240502 14:14:49.008457 26537 tablet_service.cc:1514] Processing DeleteTablet for tablet 55c42fbd424f4e7ca7f32806a891deb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:48 UTC) from {username='slave'} at 127.0.0.1:51696
I20240502 14:14:49.008813 29669 tablet_replica.cc:330] T 55c42fbd424f4e7ca7f32806a891deb2 P ba187e38a2754d60a12de9ac4856430a: stopping tablet replica
I20240502 14:14:49.008877 29423 tablet_service.cc:1514] Processing DeleteTablet for tablet 55c42fbd424f4e7ca7f32806a891deb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:48 UTC) from {username='slave'} at 127.0.0.1:40652
I20240502 14:14:49.009047 29669 raft_consensus.cc:2229] T 55c42fbd424f4e7ca7f32806a891deb2 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:49.009361 29669 raft_consensus.cc:2258] T 55c42fbd424f4e7ca7f32806a891deb2 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:49.009667 29289 tablet_service.cc:1514] Processing DeleteTablet for tablet 55c42fbd424f4e7ca7f32806a891deb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:48 UTC) from {username='slave'} at 127.0.0.1:59160
I20240502 14:14:49.010761 29669 ts_tablet_manager.cc:1875] T 55c42fbd424f4e7ca7f32806a891deb2 P ba187e38a2754d60a12de9ac4856430a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:49.021365 29669 ts_tablet_manager.cc:1888] T 55c42fbd424f4e7ca7f32806a891deb2 P ba187e38a2754d60a12de9ac4856430a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:49.023298 29669 log.cc:1196] T 55c42fbd424f4e7ca7f32806a891deb2 P ba187e38a2754d60a12de9ac4856430a: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/55c42fbd424f4e7ca7f32806a891deb2
I20240502 14:14:49.023865 29669 ts_tablet_manager.cc:1909] T 55c42fbd424f4e7ca7f32806a891deb2 P ba187e38a2754d60a12de9ac4856430a: Deleting consensus metadata
I20240502 14:14:49.025173 26271 catalog_manager.cc:4830] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): tablet 55c42fbd424f4e7ca7f32806a891deb2 (table rebalancer_dynamic_table_new_name [id=16967cf57da940a2898db0011c297b55]) successfully deleted
I20240502 14:14:49.028231 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06ae914a36b244f982cd2ace31769a79" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:14:49.028548 26426 raft_consensus.cc:2454] T 06ae914a36b244f982cd2ace31769a79 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:14:49.029019 29693 tablet_bootstrap.cc:492] T e93acdb9427943938f4c12ae2bf2c594 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:14:49.029294 29510 leader_election.cc:302] T 06ae914a36b244f982cd2ace31769a79 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:14:49.029805 29694 raft_consensus.cc:2781] T 06ae914a36b244f982cd2ace31769a79 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:49.030012 29694 raft_consensus.cc:482] T 06ae914a36b244f982cd2ace31769a79 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:49.030144 29694 raft_consensus.cc:3037] T 06ae914a36b244f982cd2ace31769a79 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:49.031176 29693 tablet_bootstrap.cc:654] T e93acdb9427943938f4c12ae2bf2c594 P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:49.033813 29694 raft_consensus.cc:504] T 06ae914a36b244f982cd2ace31769a79 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:49.034116 29693 tablet_bootstrap.cc:492] T e93acdb9427943938f4c12ae2bf2c594 P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:14:49.034263 29693 ts_tablet_manager.cc:1367] T e93acdb9427943938f4c12ae2bf2c594 P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 14:14:49.034344 29702 tablet_replica.cc:330] T 55c42fbd424f4e7ca7f32806a891deb2 P c1100e16b3b9495ca61d4f95e89b6a89: stopping tablet replica
I20240502 14:14:49.035580 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06ae914a36b244f982cd2ace31769a79" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:14:49.035594 29693 raft_consensus.cc:348] T e93acdb9427943938f4c12ae2bf2c594 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:49.035813 29693 raft_consensus.cc:374] T e93acdb9427943938f4c12ae2bf2c594 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:49.035898 29693 raft_consensus.cc:729] T e93acdb9427943938f4c12ae2bf2c594 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:14:49.035851 26426 raft_consensus.cc:3037] T 06ae914a36b244f982cd2ace31769a79 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:49.036119 29693 consensus_queue.cc:260] T e93acdb9427943938f4c12ae2bf2c594 P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:49.036661 29693 ts_tablet_manager.cc:1398] T e93acdb9427943938f4c12ae2bf2c594 P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 14:14:49.038853 26426 raft_consensus.cc:2454] T 06ae914a36b244f982cd2ace31769a79 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:14:49.039422 29694 leader_election.cc:288] T 06ae914a36b244f982cd2ace31769a79 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Requested vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:14:49.040248 29510 leader_election.cc:302] T 06ae914a36b244f982cd2ace31769a79 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:14:49.040866 29694 raft_consensus.cc:2781] T 06ae914a36b244f982cd2ace31769a79 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:49.043701 29659 raft_consensus.cc:482] T 177a34c96aa84cb4aa50bbd79cab4144 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:49.043926 29659 raft_consensus.cc:504] T 177a34c96aa84cb4aa50bbd79cab4144 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:49.044765 29659 leader_election.cc:288] T 177a34c96aa84cb4aa50bbd79cab4144 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:14:49.046239 29701 tablet_replica.cc:330] T 55c42fbd424f4e7ca7f32806a891deb2 P 68a3c2b17ed94e06aba66ed4178132ab: stopping tablet replica
I20240502 14:14:49.046133 29175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "177a34c96aa84cb4aa50bbd79cab4144" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:14:49.046588 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06ae914a36b244f982cd2ace31769a79" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:14:49.046733 29175 raft_consensus.cc:2454] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 0.
I20240502 14:14:49.046811 26818 raft_consensus.cc:2454] T 06ae914a36b244f982cd2ace31769a79 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:14:49.046937 29702 raft_consensus.cc:2229] T 55c42fbd424f4e7ca7f32806a891deb2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:49.047147 26819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06ae914a36b244f982cd2ace31769a79" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:14:49.047297 29702 raft_consensus.cc:2258] T 55c42fbd424f4e7ca7f32806a891deb2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:49.047356 26819 raft_consensus.cc:3037] T 06ae914a36b244f982cd2ace31769a79 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:49.047691 29378 leader_election.cc:302] T 177a34c96aa84cb4aa50bbd79cab4144 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 68a3c2b17ed94e06aba66ed4178132ab; no voters:
I20240502 14:14:49.047724 29701 raft_consensus.cc:2229] T 55c42fbd424f4e7ca7f32806a891deb2 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:49.048032 29659 raft_consensus.cc:2781] T 177a34c96aa84cb4aa50bbd79cab4144 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:49.048136 29659 raft_consensus.cc:482] T 177a34c96aa84cb4aa50bbd79cab4144 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:49.048238 29659 raft_consensus.cc:3037] T 177a34c96aa84cb4aa50bbd79cab4144 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:49.048465 29701 raft_consensus.cc:2258] T 55c42fbd424f4e7ca7f32806a891deb2 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:49.050424 26819 raft_consensus.cc:2454] T 06ae914a36b244f982cd2ace31769a79 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:14:49.050643 29701 ts_tablet_manager.cc:1875] T 55c42fbd424f4e7ca7f32806a891deb2 P 68a3c2b17ed94e06aba66ed4178132ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:49.051177 29659 raft_consensus.cc:504] T 177a34c96aa84cb4aa50bbd79cab4144 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:49.052525 29694 raft_consensus.cc:686] T 06ae914a36b244f982cd2ace31769a79 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Becoming Leader. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Running, Role: LEADER
I20240502 14:14:49.052994 29175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "177a34c96aa84cb4aa50bbd79cab4144" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:14:49.053237 29175 raft_consensus.cc:3037] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:49.053198 29694 consensus_queue.cc:237] T 06ae914a36b244f982cd2ace31769a79 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:49.055058 29702 ts_tablet_manager.cc:1875] T 55c42fbd424f4e7ca7f32806a891deb2 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:49.056522 29175 raft_consensus.cc:2454] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 1.
I20240502 14:14:49.057899 29378 leader_election.cc:302] T 177a34c96aa84cb4aa50bbd79cab4144 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 68a3c2b17ed94e06aba66ed4178132ab; no voters:
I20240502 14:14:49.058239 29644 raft_consensus.cc:2781] T 177a34c96aa84cb4aa50bbd79cab4144 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:49.058410 29644 raft_consensus.cc:686] T 177a34c96aa84cb4aa50bbd79cab4144 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 LEADER]: Becoming Leader. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Running, Role: LEADER
I20240502 14:14:49.058703 29644 consensus_queue.cc:237] T 177a34c96aa84cb4aa50bbd79cab4144 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:49.058993 29659 leader_election.cc:288] T 177a34c96aa84cb4aa50bbd79cab4144 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 election: Requested vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:14:49.060052 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "177a34c96aa84cb4aa50bbd79cab4144" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:14:49.060154 26423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "177a34c96aa84cb4aa50bbd79cab4144" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:14:49.060299 26426 raft_consensus.cc:2454] T 177a34c96aa84cb4aa50bbd79cab4144 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 0.
I20240502 14:14:49.075969 26283 catalog_manager.cc:5484] T 177a34c96aa84cb4aa50bbd79cab4144 P 68a3c2b17ed94e06aba66ed4178132ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199). New cstate: current_term: 1 leader_uuid: "68a3c2b17ed94e06aba66ed4178132ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:14:49.077316 29702 ts_tablet_manager.cc:1888] T 55c42fbd424f4e7ca7f32806a891deb2 P c1100e16b3b9495ca61d4f95e89b6a89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:49.077517 29702 log.cc:1196] T 55c42fbd424f4e7ca7f32806a891deb2 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal/wals/55c42fbd424f4e7ca7f32806a891deb2
I20240502 14:14:49.078131 29702 ts_tablet_manager.cc:1909] T 55c42fbd424f4e7ca7f32806a891deb2 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting consensus metadata
I20240502 14:14:49.081285 26271 catalog_manager.cc:4830] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): tablet 55c42fbd424f4e7ca7f32806a891deb2 (table rebalancer_dynamic_table_new_name [id=16967cf57da940a2898db0011c297b55]) successfully deleted
I20240502 14:14:49.088637 29701 ts_tablet_manager.cc:1888] T 55c42fbd424f4e7ca7f32806a891deb2 P 68a3c2b17ed94e06aba66ed4178132ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:49.088850 29701 log.cc:1196] T 55c42fbd424f4e7ca7f32806a891deb2 P 68a3c2b17ed94e06aba66ed4178132ab: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal/wals/55c42fbd424f4e7ca7f32806a891deb2
I20240502 14:14:49.089543 29701 ts_tablet_manager.cc:1909] T 55c42fbd424f4e7ca7f32806a891deb2 P 68a3c2b17ed94e06aba66ed4178132ab: Deleting consensus metadata
I20240502 14:14:49.089984 26283 catalog_manager.cc:5484] T 06ae914a36b244f982cd2ace31769a79 P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200). New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:14:49.091444 26272 catalog_manager.cc:4830] TS 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): tablet 55c42fbd424f4e7ca7f32806a891deb2 (table rebalancer_dynamic_table_new_name [id=16967cf57da940a2898db0011c297b55]) successfully deleted
W20240502 14:14:49.092743 29621 tablet.cc:2369] T 06ae914a36b244f982cd2ace31769a79 P 4092902b3e674857a0147f1b4f2cdb86: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:14:49.112041 26283 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:14:49.113888 26277 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240502 14:14:49.114117 26277 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240502 14:14:49.121245 26277 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet 52a4b0ee48204fd18ebb932448bd6503
I20240502 14:14:49.121896 26277 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet 18c4a357e8814da8aac5b526bc920dab
I20240502 14:14:49.122370 26277 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet dcc8fde0e9ab4236a33971bbc90716ff
I20240502 14:14:49.123668 26799 tablet_service.cc:1514] Processing DeleteTablet for tablet 18c4a357e8814da8aac5b526bc920dab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:49 UTC) from {username='slave'} at 127.0.0.1:56010
I20240502 14:14:49.123888 26407 tablet_service.cc:1514] Processing DeleteTablet for tablet 52a4b0ee48204fd18ebb932448bd6503 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:49 UTC) from {username='slave'} at 127.0.0.1:37422
I20240502 14:14:49.123981 26405 tablet_service.cc:1514] Processing DeleteTablet for tablet dcc8fde0e9ab4236a33971bbc90716ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:49 UTC) from {username='slave'} at 127.0.0.1:37422
I20240502 14:14:49.124029 26538 tablet_service.cc:1514] Processing DeleteTablet for tablet dcc8fde0e9ab4236a33971bbc90716ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:49 UTC) from {username='slave'} at 127.0.0.1:51696
I20240502 14:14:49.124110 26797 tablet_service.cc:1514] Processing DeleteTablet for tablet 52a4b0ee48204fd18ebb932448bd6503 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:49 UTC) from {username='slave'} at 127.0.0.1:56010
I20240502 14:14:49.124199 26798 tablet_service.cc:1514] Processing DeleteTablet for tablet dcc8fde0e9ab4236a33971bbc90716ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:49 UTC) from {username='slave'} at 127.0.0.1:56010
I20240502 14:14:49.124202 26668 tablet_service.cc:1514] Processing DeleteTablet for tablet 18c4a357e8814da8aac5b526bc920dab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:49 UTC) from {username='slave'} at 127.0.0.1:34150
I20240502 14:14:49.124356 29666 tablet_replica.cc:330] T 18c4a357e8814da8aac5b526bc920dab P e4cfb8fc5a474177af6848a6efa13a27: stopping tablet replica
I20240502 14:14:49.124504 29667 tablet_replica.cc:330] T 18c4a357e8814da8aac5b526bc920dab P 69ab78b2a1b74d2bbf2cdeafa0a4af23: stopping tablet replica
I20240502 14:14:49.124491 26537 tablet_service.cc:1514] Processing DeleteTablet for tablet 52a4b0ee48204fd18ebb932448bd6503 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:49 UTC) from {username='slave'} at 127.0.0.1:51696
I20240502 14:14:49.124572 29666 raft_consensus.cc:2229] T 18c4a357e8814da8aac5b526bc920dab P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:49.124722 29667 raft_consensus.cc:2229] T 18c4a357e8814da8aac5b526bc920dab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:49.124778 29666 raft_consensus.cc:2258] T 18c4a357e8814da8aac5b526bc920dab P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:49.124755 26406 tablet_service.cc:1514] Processing DeleteTablet for tablet 18c4a357e8814da8aac5b526bc920dab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:49 UTC) from {username='slave'} at 127.0.0.1:37422
I20240502 14:14:49.125010 29668 tablet_replica.cc:330] T 52a4b0ee48204fd18ebb932448bd6503 P 89af7f423cf44717a08d6b142f0ed49d: stopping tablet replica
I20240502 14:14:49.125141 29667 raft_consensus.cc:2258] T 18c4a357e8814da8aac5b526bc920dab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:49.125252 29668 raft_consensus.cc:2229] T 52a4b0ee48204fd18ebb932448bd6503 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:49.125406 29666 ts_tablet_manager.cc:1875] T 18c4a357e8814da8aac5b526bc920dab P e4cfb8fc5a474177af6848a6efa13a27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:49.125608 29668 raft_consensus.cc:2258] T 52a4b0ee48204fd18ebb932448bd6503 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:49.125893 29667 ts_tablet_manager.cc:1875] T 18c4a357e8814da8aac5b526bc920dab P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:49.126400 29668 ts_tablet_manager.cc:1875] T 52a4b0ee48204fd18ebb932448bd6503 P 89af7f423cf44717a08d6b142f0ed49d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:49.126616 29669 tablet_replica.cc:330] T dcc8fde0e9ab4236a33971bbc90716ff P ba187e38a2754d60a12de9ac4856430a: stopping tablet replica
I20240502 14:14:49.126874 29669 raft_consensus.cc:2229] T dcc8fde0e9ab4236a33971bbc90716ff P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:49.127421 29669 raft_consensus.cc:2258] T dcc8fde0e9ab4236a33971bbc90716ff P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:49.128219 29669 ts_tablet_manager.cc:1875] T dcc8fde0e9ab4236a33971bbc90716ff P ba187e38a2754d60a12de9ac4856430a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:49.132105 29666 ts_tablet_manager.cc:1888] T 18c4a357e8814da8aac5b526bc920dab P e4cfb8fc5a474177af6848a6efa13a27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:49.132248 29666 log.cc:1196] T 18c4a357e8814da8aac5b526bc920dab P e4cfb8fc5a474177af6848a6efa13a27: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/wals/18c4a357e8814da8aac5b526bc920dab
I20240502 14:14:49.132637 29667 ts_tablet_manager.cc:1888] T 18c4a357e8814da8aac5b526bc920dab P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:49.132745 29666 ts_tablet_manager.cc:1909] T 18c4a357e8814da8aac5b526bc920dab P e4cfb8fc5a474177af6848a6efa13a27: Deleting consensus metadata
I20240502 14:14:49.132782 29667 log.cc:1196] T 18c4a357e8814da8aac5b526bc920dab P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/wals/18c4a357e8814da8aac5b526bc920dab
I20240502 14:14:49.132954 29668 ts_tablet_manager.cc:1888] T 52a4b0ee48204fd18ebb932448bd6503 P 89af7f423cf44717a08d6b142f0ed49d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:49.133337 29667 ts_tablet_manager.cc:1909] T 18c4a357e8814da8aac5b526bc920dab P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting consensus metadata
I20240502 14:14:49.134480 29666 tablet_replica.cc:330] T 52a4b0ee48204fd18ebb932448bd6503 P e4cfb8fc5a474177af6848a6efa13a27: stopping tablet replica
I20240502 14:14:49.134629 26272 catalog_manager.cc:4830] TS e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): tablet 18c4a357e8814da8aac5b526bc920dab (table rebalance_test_table_2 [id=003c373a4c124a06aae79946d6d5664c]) successfully deleted
I20240502 14:14:49.133337 29668 log.cc:1196] T 52a4b0ee48204fd18ebb932448bd6503 P 89af7f423cf44717a08d6b142f0ed49d: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/52a4b0ee48204fd18ebb932448bd6503
I20240502 14:14:49.134898 29666 raft_consensus.cc:2229] T 52a4b0ee48204fd18ebb932448bd6503 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:49.135172 29666 raft_consensus.cc:2258] T 52a4b0ee48204fd18ebb932448bd6503 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:49.135464 29668 ts_tablet_manager.cc:1909] T 52a4b0ee48204fd18ebb932448bd6503 P 89af7f423cf44717a08d6b142f0ed49d: Deleting consensus metadata
I20240502 14:14:49.136317 26272 catalog_manager.cc:4830] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): tablet 18c4a357e8814da8aac5b526bc920dab (table rebalance_test_table_2 [id=003c373a4c124a06aae79946d6d5664c]) successfully deleted
I20240502 14:14:49.136468 29666 ts_tablet_manager.cc:1875] T 52a4b0ee48204fd18ebb932448bd6503 P e4cfb8fc5a474177af6848a6efa13a27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:49.136598 29668 tablet_replica.cc:330] T dcc8fde0e9ab4236a33971bbc90716ff P 89af7f423cf44717a08d6b142f0ed49d: stopping tablet replica
I20240502 14:14:49.137285 29668 raft_consensus.cc:2229] T dcc8fde0e9ab4236a33971bbc90716ff P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:49.137765 29668 raft_consensus.cc:2258] T dcc8fde0e9ab4236a33971bbc90716ff P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:49.138480 29668 ts_tablet_manager.cc:1875] T dcc8fde0e9ab4236a33971bbc90716ff P 89af7f423cf44717a08d6b142f0ed49d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:49.141907 29289 tablet_service.cc:1467] Processing CreateTablet for tablet a6ace20ae1d4481798f0a1bd4100a228 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=95cd72c4e03e430b8b16a85239e3d9c8]), partition=
I20240502 14:14:49.142354 29289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6ace20ae1d4481798f0a1bd4100a228. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:49.143153 29555 tablet_service.cc:1467] Processing CreateTablet for tablet a6ace20ae1d4481798f0a1bd4100a228 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=95cd72c4e03e430b8b16a85239e3d9c8]), partition=
I20240502 14:14:49.143512 29555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6ace20ae1d4481798f0a1bd4100a228. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:49.143940 29666 ts_tablet_manager.cc:1888] T 52a4b0ee48204fd18ebb932448bd6503 P e4cfb8fc5a474177af6848a6efa13a27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:49.146001 26271 catalog_manager.cc:4830] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): tablet 52a4b0ee48204fd18ebb932448bd6503 (table rebalance_test_table_2 [id=003c373a4c124a06aae79946d6d5664c]) successfully deleted
I20240502 14:14:49.147076 29155 tablet_service.cc:1467] Processing CreateTablet for tablet a6ace20ae1d4481798f0a1bd4100a228 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=95cd72c4e03e430b8b16a85239e3d9c8]), partition=
I20240502 14:14:49.147575 29155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6ace20ae1d4481798f0a1bd4100a228. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:49.149587 29669 ts_tablet_manager.cc:1888] T dcc8fde0e9ab4236a33971bbc90716ff P ba187e38a2754d60a12de9ac4856430a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:49.150907 29682 tablet_bootstrap.cc:492] T a6ace20ae1d4481798f0a1bd4100a228 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:14:49.151130 29669 log.cc:1196] T dcc8fde0e9ab4236a33971bbc90716ff P ba187e38a2754d60a12de9ac4856430a: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/dcc8fde0e9ab4236a33971bbc90716ff
I20240502 14:14:49.151738 29669 ts_tablet_manager.cc:1909] T dcc8fde0e9ab4236a33971bbc90716ff P ba187e38a2754d60a12de9ac4856430a: Deleting consensus metadata
I20240502 14:14:49.153214 29682 tablet_bootstrap.cc:654] T a6ace20ae1d4481798f0a1bd4100a228 P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:49.154346 26271 catalog_manager.cc:4830] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): tablet dcc8fde0e9ab4236a33971bbc90716ff (table rebalance_test_table_2 [id=003c373a4c124a06aae79946d6d5664c]) successfully deleted
I20240502 14:14:49.154995 29669 tablet_replica.cc:330] T 52a4b0ee48204fd18ebb932448bd6503 P ba187e38a2754d60a12de9ac4856430a: stopping tablet replica
I20240502 14:14:49.155844 29639 tablet_bootstrap.cc:492] T a6ace20ae1d4481798f0a1bd4100a228 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:14:49.157307 29669 raft_consensus.cc:2229] T 52a4b0ee48204fd18ebb932448bd6503 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:49.157590 29639 tablet_bootstrap.cc:654] T a6ace20ae1d4481798f0a1bd4100a228 P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:49.157637 29669 raft_consensus.cc:2258] T 52a4b0ee48204fd18ebb932448bd6503 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:49.164582 29669 ts_tablet_manager.cc:1875] T 52a4b0ee48204fd18ebb932448bd6503 P ba187e38a2754d60a12de9ac4856430a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:49.165740 29693 tablet_bootstrap.cc:492] T a6ace20ae1d4481798f0a1bd4100a228 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:14:49.167552 29693 tablet_bootstrap.cc:654] T a6ace20ae1d4481798f0a1bd4100a228 P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:49.168121 29666 log.cc:1196] T 52a4b0ee48204fd18ebb932448bd6503 P e4cfb8fc5a474177af6848a6efa13a27: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/wals/52a4b0ee48204fd18ebb932448bd6503
I20240502 14:14:49.174716 29669 ts_tablet_manager.cc:1888] T 52a4b0ee48204fd18ebb932448bd6503 P ba187e38a2754d60a12de9ac4856430a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:49.174964 29669 log.cc:1196] T 52a4b0ee48204fd18ebb932448bd6503 P ba187e38a2754d60a12de9ac4856430a: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/52a4b0ee48204fd18ebb932448bd6503
I20240502 14:14:49.175560 29669 ts_tablet_manager.cc:1909] T 52a4b0ee48204fd18ebb932448bd6503 P ba187e38a2754d60a12de9ac4856430a: Deleting consensus metadata
I20240502 14:14:49.172936 29666 ts_tablet_manager.cc:1909] T 52a4b0ee48204fd18ebb932448bd6503 P e4cfb8fc5a474177af6848a6efa13a27: Deleting consensus metadata
W20240502 14:14:49.176765 29222 tablet.cc:2369] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:14:49.177208 26271 catalog_manager.cc:4830] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): tablet 52a4b0ee48204fd18ebb932448bd6503 (table rebalance_test_table_2 [id=003c373a4c124a06aae79946d6d5664c]) successfully deleted
I20240502 14:14:49.178500 29666 tablet_replica.cc:330] T dcc8fde0e9ab4236a33971bbc90716ff P e4cfb8fc5a474177af6848a6efa13a27: stopping tablet replica
I20240502 14:14:49.178620 26272 catalog_manager.cc:4830] TS e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): tablet 52a4b0ee48204fd18ebb932448bd6503 (table rebalance_test_table_2 [id=003c373a4c124a06aae79946d6d5664c]) successfully deleted
I20240502 14:14:49.178835 29666 raft_consensus.cc:2229] T dcc8fde0e9ab4236a33971bbc90716ff P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:49.179697 29639 tablet_bootstrap.cc:492] T a6ace20ae1d4481798f0a1bd4100a228 P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:14:49.179836 29682 tablet_bootstrap.cc:492] T a6ace20ae1d4481798f0a1bd4100a228 P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:14:49.180019 29682 ts_tablet_manager.cc:1367] T a6ace20ae1d4481798f0a1bd4100a228 P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.000s
I20240502 14:14:49.180059 29639 ts_tablet_manager.cc:1367] T a6ace20ae1d4481798f0a1bd4100a228 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.003s
I20240502 14:14:49.180102 29666 raft_consensus.cc:2258] T dcc8fde0e9ab4236a33971bbc90716ff P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:49.180925 29666 ts_tablet_manager.cc:1875] T dcc8fde0e9ab4236a33971bbc90716ff P e4cfb8fc5a474177af6848a6efa13a27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:49.180920 29639 raft_consensus.cc:348] T a6ace20ae1d4481798f0a1bd4100a228 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:49.181282 29639 raft_consensus.cc:374] T a6ace20ae1d4481798f0a1bd4100a228 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:49.181288 29682 raft_consensus.cc:348] T a6ace20ae1d4481798f0a1bd4100a228 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:49.181422 29639 raft_consensus.cc:729] T a6ace20ae1d4481798f0a1bd4100a228 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:14:49.181609 29682 raft_consensus.cc:374] T a6ace20ae1d4481798f0a1bd4100a228 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:49.181761 29682 raft_consensus.cc:729] T a6ace20ae1d4481798f0a1bd4100a228 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:14:49.181710 29639 consensus_queue.cc:260] T a6ace20ae1d4481798f0a1bd4100a228 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:49.182155 29693 tablet_bootstrap.cc:492] T a6ace20ae1d4481798f0a1bd4100a228 P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:14:49.182374 29693 ts_tablet_manager.cc:1367] T a6ace20ae1d4481798f0a1bd4100a228 P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.000s
I20240502 14:14:49.183357 29682 consensus_queue.cc:260] T a6ace20ae1d4481798f0a1bd4100a228 P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:49.183502 29693 raft_consensus.cc:348] T a6ace20ae1d4481798f0a1bd4100a228 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:49.183889 29693 raft_consensus.cc:374] T a6ace20ae1d4481798f0a1bd4100a228 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:49.183974 29693 raft_consensus.cc:729] T a6ace20ae1d4481798f0a1bd4100a228 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:14:49.184302 29693 consensus_queue.cc:260] T a6ace20ae1d4481798f0a1bd4100a228 P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:49.184964 29693 ts_tablet_manager.cc:1398] T a6ace20ae1d4481798f0a1bd4100a228 P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:49.186914 29682 ts_tablet_manager.cc:1398] T a6ace20ae1d4481798f0a1bd4100a228 P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:14:49.187929 29666 ts_tablet_manager.cc:1888] T dcc8fde0e9ab4236a33971bbc90716ff P e4cfb8fc5a474177af6848a6efa13a27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:49.188153 29666 log.cc:1196] T dcc8fde0e9ab4236a33971bbc90716ff P e4cfb8fc5a474177af6848a6efa13a27: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/wals/dcc8fde0e9ab4236a33971bbc90716ff
I20240502 14:14:49.188676 29666 ts_tablet_manager.cc:1909] T dcc8fde0e9ab4236a33971bbc90716ff P e4cfb8fc5a474177af6848a6efa13a27: Deleting consensus metadata
I20240502 14:14:49.189751 29639 ts_tablet_manager.cc:1398] T a6ace20ae1d4481798f0a1bd4100a228 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240502 14:14:49.189774 26272 catalog_manager.cc:4830] TS e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): tablet dcc8fde0e9ab4236a33971bbc90716ff (table rebalance_test_table_2 [id=003c373a4c124a06aae79946d6d5664c]) successfully deleted
I20240502 14:14:49.192629 29668 ts_tablet_manager.cc:1888] T dcc8fde0e9ab4236a33971bbc90716ff P 89af7f423cf44717a08d6b142f0ed49d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:49.194128 29668 log.cc:1196] T dcc8fde0e9ab4236a33971bbc90716ff P 89af7f423cf44717a08d6b142f0ed49d: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/dcc8fde0e9ab4236a33971bbc90716ff
I20240502 14:14:49.194703 29668 ts_tablet_manager.cc:1909] T dcc8fde0e9ab4236a33971bbc90716ff P 89af7f423cf44717a08d6b142f0ed49d: Deleting consensus metadata
I20240502 14:14:49.196625 29668 tablet_replica.cc:330] T 18c4a357e8814da8aac5b526bc920dab P 89af7f423cf44717a08d6b142f0ed49d: stopping tablet replica
I20240502 14:14:49.196714 26271 catalog_manager.cc:4830] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): tablet dcc8fde0e9ab4236a33971bbc90716ff (table rebalance_test_table_2 [id=003c373a4c124a06aae79946d6d5664c]) successfully deleted
I20240502 14:14:49.198004 29668 raft_consensus.cc:2229] T 18c4a357e8814da8aac5b526bc920dab P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:49.198509 29668 raft_consensus.cc:2258] T 18c4a357e8814da8aac5b526bc920dab P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:49.199743 29668 ts_tablet_manager.cc:1875] T 18c4a357e8814da8aac5b526bc920dab P 89af7f423cf44717a08d6b142f0ed49d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:49.209936 29668 ts_tablet_manager.cc:1888] T 18c4a357e8814da8aac5b526bc920dab P 89af7f423cf44717a08d6b142f0ed49d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:49.210362 29668 log.cc:1196] T 18c4a357e8814da8aac5b526bc920dab P 89af7f423cf44717a08d6b142f0ed49d: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/18c4a357e8814da8aac5b526bc920dab
I20240502 14:14:49.212069 29668 ts_tablet_manager.cc:1909] T 18c4a357e8814da8aac5b526bc920dab P 89af7f423cf44717a08d6b142f0ed49d: Deleting consensus metadata
I20240502 14:14:49.217263 26271 catalog_manager.cc:4830] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): tablet 18c4a357e8814da8aac5b526bc920dab (table rebalance_test_table_2 [id=003c373a4c124a06aae79946d6d5664c]) successfully deleted
I20240502 14:14:49.322971 29694 raft_consensus.cc:482] T a6ace20ae1d4481798f0a1bd4100a228 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:49.323438 29694 raft_consensus.cc:504] T a6ace20ae1d4481798f0a1bd4100a228 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:49.324419 29694 leader_election.cc:288] T a6ace20ae1d4481798f0a1bd4100a228 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:14:49.332070 29175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6ace20ae1d4481798f0a1bd4100a228" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:14:49.332422 29175 raft_consensus.cc:2454] T a6ace20ae1d4481798f0a1bd4100a228 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:14:49.332823 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6ace20ae1d4481798f0a1bd4100a228" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:14:49.332876 29510 leader_election.cc:302] T a6ace20ae1d4481798f0a1bd4100a228 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 4092902b3e674857a0147f1b4f2cdb86; no voters:
I20240502 14:14:49.333062 29309 raft_consensus.cc:2454] T a6ace20ae1d4481798f0a1bd4100a228 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:14:49.333205 29694 raft_consensus.cc:2781] T a6ace20ae1d4481798f0a1bd4100a228 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:49.333319 29694 raft_consensus.cc:482] T a6ace20ae1d4481798f0a1bd4100a228 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:49.333454 29694 raft_consensus.cc:3037] T a6ace20ae1d4481798f0a1bd4100a228 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:49.335383 29694 raft_consensus.cc:504] T a6ace20ae1d4481798f0a1bd4100a228 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:49.336058 29694 leader_election.cc:288] T a6ace20ae1d4481798f0a1bd4100a228 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Requested vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:14:49.336647 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6ace20ae1d4481798f0a1bd4100a228" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
I20240502 14:14:49.336647 29175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6ace20ae1d4481798f0a1bd4100a228" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:14:49.336889 29309 raft_consensus.cc:3037] T a6ace20ae1d4481798f0a1bd4100a228 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:49.336889 29175 raft_consensus.cc:3037] T a6ace20ae1d4481798f0a1bd4100a228 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:49.340111 29175 raft_consensus.cc:2454] T a6ace20ae1d4481798f0a1bd4100a228 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:14:49.340107 29309 raft_consensus.cc:2454] T a6ace20ae1d4481798f0a1bd4100a228 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:14:49.340718 29510 leader_election.cc:302] T a6ace20ae1d4481798f0a1bd4100a228 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 4092902b3e674857a0147f1b4f2cdb86; no voters:
I20240502 14:14:49.341063 29694 raft_consensus.cc:2781] T a6ace20ae1d4481798f0a1bd4100a228 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:49.341250 29694 raft_consensus.cc:686] T a6ace20ae1d4481798f0a1bd4100a228 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Becoming Leader. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Running, Role: LEADER
I20240502 14:14:49.341568 29694 consensus_queue.cc:237] T a6ace20ae1d4481798f0a1bd4100a228 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:49.344656 26278 catalog_manager.cc:5484] T a6ace20ae1d4481798f0a1bd4100a228 P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200). New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:14:49.349807 29175 raft_consensus.cc:3037] T e93acdb9427943938f4c12ae2bf2c594 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:49.355600 29644 consensus_queue.cc:1035] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:49.398778 29652 consensus_queue.cc:1035] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:49.415972 29155 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:14:49.422250 29423 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:49.424124 29555 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:49.452206 26278 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:14:49.462424 29644 consensus_queue.cc:1035] T 177a34c96aa84cb4aa50bbd79cab4144 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:49.471889 29694 consensus_queue.cc:1035] T 06ae914a36b244f982cd2ace31769a79 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:49.479758 26406 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:14:49.498489 29289 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:14:49.502792 26537 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:14:49.536448 29309 raft_consensus.cc:1261] T a6ace20ae1d4481798f0a1bd4100a228 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:49.536478 29176 raft_consensus.cc:1261] T a6ace20ae1d4481798f0a1bd4100a228 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:49.537453 29704 consensus_queue.cc:1035] T a6ace20ae1d4481798f0a1bd4100a228 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:49.538129 29704 consensus_queue.cc:1035] T a6ace20ae1d4481798f0a1bd4100a228 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:49.548779 26424 raft_consensus.cc:3037] T 177a34c96aa84cb4aa50bbd79cab4144 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:49.558182 26668 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:14:49.561249 29644 consensus_queue.cc:1035] T 177a34c96aa84cb4aa50bbd79cab4144 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:49.592836 29704 consensus_queue.cc:1035] T 06ae914a36b244f982cd2ace31769a79 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:49.601668 29752 mvcc.cc:205] Tried to move back new op lower bound from 7023244449932500992 to 7023244449146191872. Current Snapshot: MvccSnapshot[applied={T|T < 7023244449932500992}]
I20240502 14:14:49.604897 26284 catalog_manager.cc:5576] TS 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715) does not have the latest schema for tablet a6ace20ae1d4481798f0a1bd4100a228 (table rebalancer_dynamic_table [id=95cd72c4e03e430b8b16a85239e3d9c8]). Expected version 1 got 0
I20240502 14:14:49.612027 26278 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_extra_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:49.626982 26284 catalog_manager.cc:5576] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659) does not have the latest schema for tablet a6ace20ae1d4481798f0a1bd4100a228 (table rebalancer_dynamic_table [id=95cd72c4e03e430b8b16a85239e3d9c8]). Expected version 1 got 0
I20240502 14:14:49.626250 29771 tablet.cc:1722] T a6ace20ae1d4481798f0a1bd4100a228 P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:49.628377 29771 tablet.cc:1620] T a6ace20ae1d4481798f0a1bd4100a228 P c1100e16b3b9495ca61d4f95e89b6a89: MemRowSet was empty: no flush needed.
I20240502 14:14:49.635345 26282 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240502 14:14:49.635622 26282 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240502 14:14:49.638396 26284 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=95cd72c4e03e430b8b16a85239e3d9c8] alter complete (version 1)
I20240502 14:14:49.666500 29768 tablet.cc:1722] T a6ace20ae1d4481798f0a1bd4100a228 P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:49.668541 29769 tablet.cc:1722] T a6ace20ae1d4481798f0a1bd4100a228 P 1cb67158133e4648a6eceba32d4dbba3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:49.674695 29289 tablet_service.cc:1467] Processing CreateTablet for tablet abdd0f01d8d5453793c7e9461af10516 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=df45ec00aaba41b2b3f62c6fb9633ae0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:49.675256 29289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdd0f01d8d5453793c7e9461af10516. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:49.677320 29555 tablet_service.cc:1467] Processing CreateTablet for tablet f8923ed1ce94441a8eb303b6f7dd3acb (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=df45ec00aaba41b2b3f62c6fb9633ae0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:49.677485 29554 tablet_service.cc:1467] Processing CreateTablet for tablet abdd0f01d8d5453793c7e9461af10516 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=df45ec00aaba41b2b3f62c6fb9633ae0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:49.677839 29555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8923ed1ce94441a8eb303b6f7dd3acb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:49.678174 29554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdd0f01d8d5453793c7e9461af10516. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:49.678797 29553 tablet_service.cc:1467] Processing CreateTablet for tablet 0c7ccae60f58427fbd39c26b98817963 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=df45ec00aaba41b2b3f62c6fb9633ae0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:49.679261 29553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c7ccae60f58427fbd39c26b98817963. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:49.683373 29639 tablet_bootstrap.cc:492] T abdd0f01d8d5453793c7e9461af10516 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:14:49.684911 29423 tablet_service.cc:1467] Processing CreateTablet for tablet f8923ed1ce94441a8eb303b6f7dd3acb (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=df45ec00aaba41b2b3f62c6fb9633ae0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:49.685427 29423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8923ed1ce94441a8eb303b6f7dd3acb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:49.688897 29421 tablet_service.cc:1467] Processing CreateTablet for tablet abdd0f01d8d5453793c7e9461af10516 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=df45ec00aaba41b2b3f62c6fb9633ae0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:49.689344 29421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdd0f01d8d5453793c7e9461af10516. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:49.692890 29422 tablet_service.cc:1467] Processing CreateTablet for tablet 0c7ccae60f58427fbd39c26b98817963 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=df45ec00aaba41b2b3f62c6fb9633ae0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:49.693440 26278 catalog_manager.cc:5576] TS 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715) does not have the latest schema for tablet a6ace20ae1d4481798f0a1bd4100a228 (table rebalancer_dynamic_table [id=95cd72c4e03e430b8b16a85239e3d9c8]). Expected version 1 got 0
I20240502 14:14:49.695087 26668 tablet_service.cc:1467] Processing CreateTablet for tablet f8923ed1ce94441a8eb303b6f7dd3acb (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=df45ec00aaba41b2b3f62c6fb9633ae0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:49.695624 26668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8923ed1ce94441a8eb303b6f7dd3acb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:49.700846 26669 tablet_service.cc:1467] Processing CreateTablet for tablet 0c7ccae60f58427fbd39c26b98817963 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=df45ec00aaba41b2b3f62c6fb9633ae0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:49.701313 26669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c7ccae60f58427fbd39c26b98817963. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:49.706676 29639 tablet_bootstrap.cc:654] T abdd0f01d8d5453793c7e9461af10516 P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:49.710038 29768 tablet.cc:1620] T a6ace20ae1d4481798f0a1bd4100a228 P 4092902b3e674857a0147f1b4f2cdb86: MemRowSet was empty: no flush needed.
I20240502 14:14:49.709609 29769 tablet.cc:1620] T a6ace20ae1d4481798f0a1bd4100a228 P 1cb67158133e4648a6eceba32d4dbba3: MemRowSet was empty: no flush needed.
I20240502 14:14:49.722379 29422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c7ccae60f58427fbd39c26b98817963. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:49.741531 29776 tablet_bootstrap.cc:492] T f8923ed1ce94441a8eb303b6f7dd3acb P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:49.745712 29777 tablet_bootstrap.cc:492] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:14:49.750176 29777 tablet_bootstrap.cc:654] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:49.751524 29776 tablet_bootstrap.cc:654] T f8923ed1ce94441a8eb303b6f7dd3acb P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:49.760977 26282 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet a7aae43aeafc491b84d879fecfbd983f
I20240502 14:14:49.761636 26282 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet 8a8bd7225f5c48feb980049b99eb99c1
I20240502 14:14:49.761724 26278 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:14:49.762070 26282 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet b8dd27ac333542a4844172fdc8e9b9bd
I20240502 14:14:49.762284 26537 tablet_service.cc:1514] Processing DeleteTablet for tablet a7aae43aeafc491b84d879fecfbd983f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:49 UTC) from {username='slave'} at 127.0.0.1:51696
I20240502 14:14:49.763335 26406 tablet_service.cc:1514] Processing DeleteTablet for tablet a7aae43aeafc491b84d879fecfbd983f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:49 UTC) from {username='slave'} at 127.0.0.1:37422
I20240502 14:14:49.763619 26797 tablet_service.cc:1514] Processing DeleteTablet for tablet b8dd27ac333542a4844172fdc8e9b9bd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:49 UTC) from {username='slave'} at 127.0.0.1:56010
I20240502 14:14:49.763768 26798 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a8bd7225f5c48feb980049b99eb99c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:49 UTC) from {username='slave'} at 127.0.0.1:56010
I20240502 14:14:49.763886 26669 tablet_service.cc:1514] Processing DeleteTablet for tablet b8dd27ac333542a4844172fdc8e9b9bd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:49 UTC) from {username='slave'} at 127.0.0.1:34150
I20240502 14:14:49.764329 26668 tablet_service.cc:1514] Processing DeleteTablet for tablet a7aae43aeafc491b84d879fecfbd983f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:49 UTC) from {username='slave'} at 127.0.0.1:34150
I20240502 14:14:49.764364 26538 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a8bd7225f5c48feb980049b99eb99c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:49 UTC) from {username='slave'} at 127.0.0.1:51696
I20240502 14:14:49.765237 26405 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a8bd7225f5c48feb980049b99eb99c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:49 UTC) from {username='slave'} at 127.0.0.1:37422
I20240502 14:14:49.765501 26407 tablet_service.cc:1514] Processing DeleteTablet for tablet b8dd27ac333542a4844172fdc8e9b9bd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:49 UTC) from {username='slave'} at 127.0.0.1:37422
I20240502 14:14:49.777491 29775 tablet_bootstrap.cc:492] T abdd0f01d8d5453793c7e9461af10516 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:14:49.780097 29784 tablet_replica.cc:330] T b8dd27ac333542a4844172fdc8e9b9bd P 69ab78b2a1b74d2bbf2cdeafa0a4af23: stopping tablet replica
I20240502 14:14:49.780449 29784 raft_consensus.cc:2229] T b8dd27ac333542a4844172fdc8e9b9bd P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:49.780819 29784 raft_consensus.cc:2258] T b8dd27ac333542a4844172fdc8e9b9bd P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:49.781653 29784 ts_tablet_manager.cc:1875] T b8dd27ac333542a4844172fdc8e9b9bd P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:49.785472 29639 tablet_bootstrap.cc:492] T abdd0f01d8d5453793c7e9461af10516 P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:14:49.785730 29639 ts_tablet_manager.cc:1367] T abdd0f01d8d5453793c7e9461af10516 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.102s user 0.009s sys 0.004s
I20240502 14:14:49.786670 29639 raft_consensus.cc:348] T abdd0f01d8d5453793c7e9461af10516 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:49.787066 29639 raft_consensus.cc:374] T abdd0f01d8d5453793c7e9461af10516 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:49.787247 29639 raft_consensus.cc:729] T abdd0f01d8d5453793c7e9461af10516 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:14:49.787510 29771 tablet.cc:1722] T a6ace20ae1d4481798f0a1bd4100a228 P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:49.787613 29639 consensus_queue.cc:260] T abdd0f01d8d5453793c7e9461af10516 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:49.788169 29771 tablet.cc:1620] T a6ace20ae1d4481798f0a1bd4100a228 P c1100e16b3b9495ca61d4f95e89b6a89: MemRowSet was empty: no flush needed.
I20240502 14:14:49.790205 29781 tablet_replica.cc:330] T a7aae43aeafc491b84d879fecfbd983f P ba187e38a2754d60a12de9ac4856430a: stopping tablet replica
I20240502 14:14:49.790684 29782 tablet_replica.cc:330] T a7aae43aeafc491b84d879fecfbd983f P 89af7f423cf44717a08d6b142f0ed49d: stopping tablet replica
I20240502 14:14:49.797083 29783 tablet_replica.cc:330] T b8dd27ac333542a4844172fdc8e9b9bd P e4cfb8fc5a474177af6848a6efa13a27: stopping tablet replica
I20240502 14:14:49.797344 29782 raft_consensus.cc:2229] T a7aae43aeafc491b84d879fecfbd983f P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:49.797729 29781 raft_consensus.cc:2229] T a7aae43aeafc491b84d879fecfbd983f P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:49.798332 29783 raft_consensus.cc:2229] T b8dd27ac333542a4844172fdc8e9b9bd P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:49.798691 29782 raft_consensus.cc:2258] T a7aae43aeafc491b84d879fecfbd983f P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:49.799057 29781 raft_consensus.cc:2258] T a7aae43aeafc491b84d879fecfbd983f P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:49.799404 29783 raft_consensus.cc:2258] T b8dd27ac333542a4844172fdc8e9b9bd P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:49.802937 29781 ts_tablet_manager.cc:1875] T a7aae43aeafc491b84d879fecfbd983f P ba187e38a2754d60a12de9ac4856430a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:49.803812 29783 ts_tablet_manager.cc:1875] T b8dd27ac333542a4844172fdc8e9b9bd P e4cfb8fc5a474177af6848a6efa13a27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:49.804466 29782 ts_tablet_manager.cc:1875] T a7aae43aeafc491b84d879fecfbd983f P 89af7f423cf44717a08d6b142f0ed49d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:49.805562 29775 tablet_bootstrap.cc:654] T abdd0f01d8d5453793c7e9461af10516 P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:49.806087 29639 ts_tablet_manager.cc:1398] T abdd0f01d8d5453793c7e9461af10516 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20240502 14:14:49.807058 26278 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=95cd72c4e03e430b8b16a85239e3d9c8] alter complete (version 2)
I20240502 14:14:49.812145 29769 tablet.cc:1722] T a6ace20ae1d4481798f0a1bd4100a228 P 1cb67158133e4648a6eceba32d4dbba3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:49.812762 29769 tablet.cc:1620] T a6ace20ae1d4481798f0a1bd4100a228 P 1cb67158133e4648a6eceba32d4dbba3: MemRowSet was empty: no flush needed.
I20240502 14:14:49.813482 29768 tablet.cc:1722] T a6ace20ae1d4481798f0a1bd4100a228 P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
W20240502 14:14:49.814275 26363 consensus_peers.cc:485] T b8dd27ac333542a4844172fdc8e9b9bd P 89af7f423cf44717a08d6b142f0ed49d -> Peer e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): Couldn't send request to peer e4cfb8fc5a474177af6848a6efa13a27. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:49.815872 29768 tablet.cc:1620] T a6ace20ae1d4481798f0a1bd4100a228 P 4092902b3e674857a0147f1b4f2cdb86: MemRowSet was empty: no flush needed.
I20240502 14:14:49.834563 29782 ts_tablet_manager.cc:1888] T a7aae43aeafc491b84d879fecfbd983f P 89af7f423cf44717a08d6b142f0ed49d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:49.841706 29775 tablet_bootstrap.cc:492] T abdd0f01d8d5453793c7e9461af10516 P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:14:49.841964 29775 ts_tablet_manager.cc:1367] T abdd0f01d8d5453793c7e9461af10516 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.065s user 0.002s sys 0.003s
I20240502 14:14:49.843015 29775 raft_consensus.cc:348] T abdd0f01d8d5453793c7e9461af10516 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:49.843286 29775 raft_consensus.cc:374] T abdd0f01d8d5453793c7e9461af10516 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:49.843389 29775 raft_consensus.cc:729] T abdd0f01d8d5453793c7e9461af10516 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:14:49.843726 29775 consensus_queue.cc:260] T abdd0f01d8d5453793c7e9461af10516 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:49.845297 29775 ts_tablet_manager.cc:1398] T abdd0f01d8d5453793c7e9461af10516 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:14:49.845692 29775 tablet_bootstrap.cc:492] T f8923ed1ce94441a8eb303b6f7dd3acb P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:14:49.848323 29777 tablet_bootstrap.cc:492] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:14:49.848604 29777 ts_tablet_manager.cc:1367] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.103s user 0.005s sys 0.017s
I20240502 14:14:49.849581 29777 raft_consensus.cc:348] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:49.849885 29777 raft_consensus.cc:374] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:49.850032 29777 raft_consensus.cc:729] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:14:49.850440 29777 consensus_queue.cc:260] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:49.851241 29777 ts_tablet_manager.cc:1398] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:14:49.851704 29777 tablet_bootstrap.cc:492] T abdd0f01d8d5453793c7e9461af10516 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:14:49.853099 29776 tablet_bootstrap.cc:492] T f8923ed1ce94441a8eb303b6f7dd3acb P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:49.853315 29776 ts_tablet_manager.cc:1367] T f8923ed1ce94441a8eb303b6f7dd3acb P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.112s user 0.002s sys 0.020s
I20240502 14:14:49.853593 29777 tablet_bootstrap.cc:654] T abdd0f01d8d5453793c7e9461af10516 P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:49.854269 29776 raft_consensus.cc:348] T f8923ed1ce94441a8eb303b6f7dd3acb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:49.854564 29776 raft_consensus.cc:374] T f8923ed1ce94441a8eb303b6f7dd3acb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:49.854678 29776 raft_consensus.cc:729] T f8923ed1ce94441a8eb303b6f7dd3acb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:49.854941 29776 consensus_queue.cc:260] T f8923ed1ce94441a8eb303b6f7dd3acb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:49.855633 29776 ts_tablet_manager.cc:1398] T f8923ed1ce94441a8eb303b6f7dd3acb P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:14:49.856020 29776 tablet_bootstrap.cc:492] T 0c7ccae60f58427fbd39c26b98817963 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:49.858083 29776 tablet_bootstrap.cc:654] T 0c7ccae60f58427fbd39c26b98817963 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:49.862386 26797 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:14:49.876257 29783 ts_tablet_manager.cc:1888] T b8dd27ac333542a4844172fdc8e9b9bd P e4cfb8fc5a474177af6848a6efa13a27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:49.889968 29775 tablet_bootstrap.cc:654] T f8923ed1ce94441a8eb303b6f7dd3acb P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:49.892241 29784 ts_tablet_manager.cc:1888] T b8dd27ac333542a4844172fdc8e9b9bd P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:49.892503 29784 log.cc:1196] T b8dd27ac333542a4844172fdc8e9b9bd P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/wals/b8dd27ac333542a4844172fdc8e9b9bd
I20240502 14:14:49.893030 29784 ts_tablet_manager.cc:1909] T b8dd27ac333542a4844172fdc8e9b9bd P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting consensus metadata
I20240502 14:14:49.894447 29784 tablet_replica.cc:330] T a7aae43aeafc491b84d879fecfbd983f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: stopping tablet replica
I20240502 14:14:49.894739 29784 raft_consensus.cc:2229] T a7aae43aeafc491b84d879fecfbd983f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:49.895061 26272 catalog_manager.cc:4830] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): tablet b8dd27ac333542a4844172fdc8e9b9bd (table rebalance_test_table_4 [id=10659483777045cb826c0f72b9086a71]) successfully deleted
I20240502 14:14:49.907606 29704 raft_consensus.cc:482] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:49.907847 29704 raft_consensus.cc:504] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:49.908932 29704 leader_election.cc:288] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:14:49.895165 29784 raft_consensus.cc:2258] T a7aae43aeafc491b84d879fecfbd983f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:49.912887 29784 ts_tablet_manager.cc:1875] T a7aae43aeafc491b84d879fecfbd983f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:49.912967 26277 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:14:49.916002 29781 ts_tablet_manager.cc:1888] T a7aae43aeafc491b84d879fecfbd983f P ba187e38a2754d60a12de9ac4856430a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:49.886554 29782 log.cc:1196] T a7aae43aeafc491b84d879fecfbd983f P 89af7f423cf44717a08d6b142f0ed49d: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/a7aae43aeafc491b84d879fecfbd983f
I20240502 14:14:49.917181 29782 ts_tablet_manager.cc:1909] T a7aae43aeafc491b84d879fecfbd983f P 89af7f423cf44717a08d6b142f0ed49d: Deleting consensus metadata
I20240502 14:14:49.917701 29783 log.cc:1196] T b8dd27ac333542a4844172fdc8e9b9bd P e4cfb8fc5a474177af6848a6efa13a27: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/wals/b8dd27ac333542a4844172fdc8e9b9bd
I20240502 14:14:49.918503 29781 log.cc:1196] T a7aae43aeafc491b84d879fecfbd983f P ba187e38a2754d60a12de9ac4856430a: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/a7aae43aeafc491b84d879fecfbd983f
I20240502 14:14:49.919152 29783 ts_tablet_manager.cc:1909] T b8dd27ac333542a4844172fdc8e9b9bd P e4cfb8fc5a474177af6848a6efa13a27: Deleting consensus metadata
I20240502 14:14:49.920032 29781 ts_tablet_manager.cc:1909] T a7aae43aeafc491b84d879fecfbd983f P ba187e38a2754d60a12de9ac4856430a: Deleting consensus metadata
I20240502 14:14:49.931449 29771 tablet.cc:1722] T a6ace20ae1d4481798f0a1bd4100a228 P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:49.932185 26271 catalog_manager.cc:4830] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): tablet a7aae43aeafc491b84d879fecfbd983f (table rebalance_test_table_4 [id=10659483777045cb826c0f72b9086a71]) successfully deleted
I20240502 14:14:49.935254 29769 tablet.cc:1722] T a6ace20ae1d4481798f0a1bd4100a228 P 1cb67158133e4648a6eceba32d4dbba3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:49.936820 26272 catalog_manager.cc:4830] TS e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): tablet b8dd27ac333542a4844172fdc8e9b9bd (table rebalance_test_table_4 [id=10659483777045cb826c0f72b9086a71]) successfully deleted
I20240502 14:14:49.927910 29782 tablet_replica.cc:330] T 8a8bd7225f5c48feb980049b99eb99c1 P 89af7f423cf44717a08d6b142f0ed49d: stopping tablet replica
I20240502 14:14:49.937860 29782 raft_consensus.cc:2229] T 8a8bd7225f5c48feb980049b99eb99c1 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:49.938304 29782 raft_consensus.cc:2258] T 8a8bd7225f5c48feb980049b99eb99c1 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:49.938709 29768 tablet.cc:1722] T a6ace20ae1d4481798f0a1bd4100a228 P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:49.941219 26277 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table_new_name [id=95cd72c4e03e430b8b16a85239e3d9c8] alter complete (version 3)
I20240502 14:14:49.947273 26271 catalog_manager.cc:4830] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): tablet a7aae43aeafc491b84d879fecfbd983f (table rebalance_test_table_4 [id=10659483777045cb826c0f72b9086a71]) successfully deleted
I20240502 14:14:49.948913 29781 tablet_replica.cc:330] T 8a8bd7225f5c48feb980049b99eb99c1 P ba187e38a2754d60a12de9ac4856430a: stopping tablet replica
I20240502 14:14:49.949369 29781 raft_consensus.cc:2229] T 8a8bd7225f5c48feb980049b99eb99c1 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:49.949800 29781 raft_consensus.cc:2258] T 8a8bd7225f5c48feb980049b99eb99c1 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:49.950982 29781 ts_tablet_manager.cc:1875] T 8a8bd7225f5c48feb980049b99eb99c1 P ba187e38a2754d60a12de9ac4856430a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:49.951727 29776 tablet_bootstrap.cc:492] T 0c7ccae60f58427fbd39c26b98817963 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:49.952311 29776 ts_tablet_manager.cc:1367] T 0c7ccae60f58427fbd39c26b98817963 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.096s user 0.004s sys 0.000s
I20240502 14:14:49.954113 29783 tablet_replica.cc:330] T 8a8bd7225f5c48feb980049b99eb99c1 P e4cfb8fc5a474177af6848a6efa13a27: stopping tablet replica
I20240502 14:14:49.954439 29783 raft_consensus.cc:2229] T 8a8bd7225f5c48feb980049b99eb99c1 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:49.954489 29776 raft_consensus.cc:348] T 0c7ccae60f58427fbd39c26b98817963 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:49.955487 29775 tablet_bootstrap.cc:492] T f8923ed1ce94441a8eb303b6f7dd3acb P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:14:49.956012 29776 raft_consensus.cc:374] T 0c7ccae60f58427fbd39c26b98817963 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:49.956207 29776 raft_consensus.cc:729] T 0c7ccae60f58427fbd39c26b98817963 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:49.956681 29776 consensus_queue.cc:260] T 0c7ccae60f58427fbd39c26b98817963 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:49.955821 29784 ts_tablet_manager.cc:1888] T a7aae43aeafc491b84d879fecfbd983f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:49.957248 29784 log.cc:1196] T a7aae43aeafc491b84d879fecfbd983f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/wals/a7aae43aeafc491b84d879fecfbd983f
I20240502 14:14:49.957571 29775 ts_tablet_manager.cc:1367] T f8923ed1ce94441a8eb303b6f7dd3acb P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.110s user 0.003s sys 0.003s
I20240502 14:14:49.959098 29784 ts_tablet_manager.cc:1909] T a7aae43aeafc491b84d879fecfbd983f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting consensus metadata
I20240502 14:14:49.959422 29775 raft_consensus.cc:348] T f8923ed1ce94441a8eb303b6f7dd3acb P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:49.959878 29775 raft_consensus.cc:374] T f8923ed1ce94441a8eb303b6f7dd3acb P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:49.960029 29775 raft_consensus.cc:729] T f8923ed1ce94441a8eb303b6f7dd3acb P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:14:49.960515 29775 consensus_queue.cc:260] T f8923ed1ce94441a8eb303b6f7dd3acb P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:49.961350 29777 tablet_bootstrap.cc:492] T abdd0f01d8d5453793c7e9461af10516 P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:14:49.961321 29775 ts_tablet_manager.cc:1398] T f8923ed1ce94441a8eb303b6f7dd3acb P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:14:49.961544 29777 ts_tablet_manager.cc:1367] T abdd0f01d8d5453793c7e9461af10516 P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.110s user 0.039s sys 0.034s
I20240502 14:14:49.961846 29775 tablet_bootstrap.cc:492] T 0c7ccae60f58427fbd39c26b98817963 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:14:49.959686 29776 ts_tablet_manager.cc:1398] T 0c7ccae60f58427fbd39c26b98817963 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20240502 14:14:49.962437 29777 raft_consensus.cc:348] T abdd0f01d8d5453793c7e9461af10516 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:49.962689 29777 raft_consensus.cc:374] T abdd0f01d8d5453793c7e9461af10516 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:49.962775 29777 raft_consensus.cc:729] T abdd0f01d8d5453793c7e9461af10516 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:14:49.962842 26272 catalog_manager.cc:4830] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): tablet a7aae43aeafc491b84d879fecfbd983f (table rebalance_test_table_4 [id=10659483777045cb826c0f72b9086a71]) successfully deleted
I20240502 14:14:49.963052 29777 consensus_queue.cc:260] T abdd0f01d8d5453793c7e9461af10516 P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:49.964638 29777 ts_tablet_manager.cc:1398] T abdd0f01d8d5453793c7e9461af10516 P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 14:14:49.964965 29777 tablet_bootstrap.cc:492] T f8923ed1ce94441a8eb303b6f7dd3acb P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:14:49.967775 29783 raft_consensus.cc:2258] T 8a8bd7225f5c48feb980049b99eb99c1 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:49.968458 29775 tablet_bootstrap.cc:654] T 0c7ccae60f58427fbd39c26b98817963 P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:49.972118 29777 tablet_bootstrap.cc:654] T f8923ed1ce94441a8eb303b6f7dd3acb P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:49.972597 29783 ts_tablet_manager.cc:1875] T 8a8bd7225f5c48feb980049b99eb99c1 P e4cfb8fc5a474177af6848a6efa13a27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:49.972859 29782 ts_tablet_manager.cc:1875] T 8a8bd7225f5c48feb980049b99eb99c1 P 89af7f423cf44717a08d6b142f0ed49d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:49.982142 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c7ccae60f58427fbd39c26b98817963" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
W20240502 14:14:49.983422 29511 leader_election.cc:341] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:49.992602 29775 tablet_bootstrap.cc:492] T 0c7ccae60f58427fbd39c26b98817963 P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:14:49.992823 29775 ts_tablet_manager.cc:1367] T 0c7ccae60f58427fbd39c26b98817963 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.000s
I20240502 14:14:49.993892 29775 raft_consensus.cc:348] T 0c7ccae60f58427fbd39c26b98817963 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:49.994204 29775 raft_consensus.cc:374] T 0c7ccae60f58427fbd39c26b98817963 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:49.994321 29775 raft_consensus.cc:729] T 0c7ccae60f58427fbd39c26b98817963 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:14:49.994663 29775 consensus_queue.cc:260] T 0c7ccae60f58427fbd39c26b98817963 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:49.996851 29775 ts_tablet_manager.cc:1398] T 0c7ccae60f58427fbd39c26b98817963 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:14:50.004724 26688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c7ccae60f58427fbd39c26b98817963" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:14:50.004998 26688 raft_consensus.cc:2454] T 0c7ccae60f58427fbd39c26b98817963 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:14:50.005484 29511 leader_election.cc:302] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters: 68a3c2b17ed94e06aba66ed4178132ab
I20240502 14:14:50.005849 29704 raft_consensus.cc:2781] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:50.005959 29704 raft_consensus.cc:482] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:50.006074 29704 raft_consensus.cc:3037] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:50.007048 29782 ts_tablet_manager.cc:1888] T 8a8bd7225f5c48feb980049b99eb99c1 P 89af7f423cf44717a08d6b142f0ed49d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:50.007200 29782 log.cc:1196] T 8a8bd7225f5c48feb980049b99eb99c1 P 89af7f423cf44717a08d6b142f0ed49d: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/8a8bd7225f5c48feb980049b99eb99c1
I20240502 14:14:50.007669 29782 ts_tablet_manager.cc:1909] T 8a8bd7225f5c48feb980049b99eb99c1 P 89af7f423cf44717a08d6b142f0ed49d: Deleting consensus metadata
I20240502 14:14:50.008773 29782 tablet_replica.cc:330] T b8dd27ac333542a4844172fdc8e9b9bd P 89af7f423cf44717a08d6b142f0ed49d: stopping tablet replica
I20240502 14:14:50.008986 29782 raft_consensus.cc:2229] T b8dd27ac333542a4844172fdc8e9b9bd P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:50.008968 29704 raft_consensus.cc:504] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:50.009343 29782 raft_consensus.cc:2258] T b8dd27ac333542a4844172fdc8e9b9bd P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:50.009416 29777 tablet_bootstrap.cc:492] T f8923ed1ce94441a8eb303b6f7dd3acb P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:14:50.009605 29777 ts_tablet_manager.cc:1367] T f8923ed1ce94441a8eb303b6f7dd3acb P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.045s user 0.015s sys 0.009s
I20240502 14:14:50.009748 29704 leader_election.cc:288] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Requested vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:14:50.010116 29782 ts_tablet_manager.cc:1875] T b8dd27ac333542a4844172fdc8e9b9bd P 89af7f423cf44717a08d6b142f0ed49d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:50.010259 26271 catalog_manager.cc:4830] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): tablet 8a8bd7225f5c48feb980049b99eb99c1 (table rebalance_test_table_4 [id=10659483777045cb826c0f72b9086a71]) successfully deleted
I20240502 14:14:50.010593 29777 raft_consensus.cc:348] T f8923ed1ce94441a8eb303b6f7dd3acb P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:50.010807 29777 raft_consensus.cc:374] T f8923ed1ce94441a8eb303b6f7dd3acb P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:50.010886 29777 raft_consensus.cc:729] T f8923ed1ce94441a8eb303b6f7dd3acb P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:14:50.011128 29777 consensus_queue.cc:260] T f8923ed1ce94441a8eb303b6f7dd3acb P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:50.011672 29777 ts_tablet_manager.cc:1398] T f8923ed1ce94441a8eb303b6f7dd3acb P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:14:50.011677 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c7ccae60f58427fbd39c26b98817963" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
I20240502 14:14:50.011938 29443 raft_consensus.cc:3037] T 0c7ccae60f58427fbd39c26b98817963 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:50.015175 29443 raft_consensus.cc:2454] T 0c7ccae60f58427fbd39c26b98817963 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:14:50.015938 29781 ts_tablet_manager.cc:1888] T 8a8bd7225f5c48feb980049b99eb99c1 P ba187e38a2754d60a12de9ac4856430a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:50.016183 29511 leader_election.cc:302] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, 68a3c2b17ed94e06aba66ed4178132ab; no voters:
I20240502 14:14:50.016527 29783 ts_tablet_manager.cc:1888] T 8a8bd7225f5c48feb980049b99eb99c1 P e4cfb8fc5a474177af6848a6efa13a27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:50.016527 26688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c7ccae60f58427fbd39c26b98817963" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:14:50.016690 29694 raft_consensus.cc:2781] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:50.016781 26688 raft_consensus.cc:3037] T 0c7ccae60f58427fbd39c26b98817963 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:50.016899 29694 raft_consensus.cc:686] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Becoming Leader. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Running, Role: LEADER
I20240502 14:14:50.017052 29783 log.cc:1196] T 8a8bd7225f5c48feb980049b99eb99c1 P e4cfb8fc5a474177af6848a6efa13a27: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/wals/8a8bd7225f5c48feb980049b99eb99c1
I20240502 14:14:50.017232 29694 consensus_queue.cc:237] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:50.017850 29783 ts_tablet_manager.cc:1909] T 8a8bd7225f5c48feb980049b99eb99c1 P e4cfb8fc5a474177af6848a6efa13a27: Deleting consensus metadata
I20240502 14:14:50.018494 29782 ts_tablet_manager.cc:1888] T b8dd27ac333542a4844172fdc8e9b9bd P 89af7f423cf44717a08d6b142f0ed49d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:50.018751 29782 log.cc:1196] T b8dd27ac333542a4844172fdc8e9b9bd P 89af7f423cf44717a08d6b142f0ed49d: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/b8dd27ac333542a4844172fdc8e9b9bd
I20240502 14:14:50.019053 29781 log.cc:1196] T 8a8bd7225f5c48feb980049b99eb99c1 P ba187e38a2754d60a12de9ac4856430a: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/8a8bd7225f5c48feb980049b99eb99c1
I20240502 14:14:50.019332 29782 ts_tablet_manager.cc:1909] T b8dd27ac333542a4844172fdc8e9b9bd P 89af7f423cf44717a08d6b142f0ed49d: Deleting consensus metadata
I20240502 14:14:50.019351 26272 catalog_manager.cc:4830] TS e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): tablet 8a8bd7225f5c48feb980049b99eb99c1 (table rebalance_test_table_4 [id=10659483777045cb826c0f72b9086a71]) successfully deleted
I20240502 14:14:50.022044 26688 raft_consensus.cc:2454] T 0c7ccae60f58427fbd39c26b98817963 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:14:50.022035 26271 catalog_manager.cc:4830] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): tablet b8dd27ac333542a4844172fdc8e9b9bd (table rebalance_test_table_4 [id=10659483777045cb826c0f72b9086a71]) successfully deleted
I20240502 14:14:50.026492 29781 ts_tablet_manager.cc:1909] T 8a8bd7225f5c48feb980049b99eb99c1 P ba187e38a2754d60a12de9ac4856430a: Deleting consensus metadata
I20240502 14:14:50.028146 26271 catalog_manager.cc:4830] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): tablet 8a8bd7225f5c48feb980049b99eb99c1 (table rebalance_test_table_4 [id=10659483777045cb826c0f72b9086a71]) successfully deleted
I20240502 14:14:50.034812 26278 catalog_manager.cc:5484] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200). New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:14:50.045279 26278 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:14:50.045564 26278 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:14:50.049495 26278 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet a6ace20ae1d4481798f0a1bd4100a228
I20240502 14:14:50.050571 29289 tablet_service.cc:1514] Processing DeleteTablet for tablet a6ace20ae1d4481798f0a1bd4100a228 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:59160
I20240502 14:14:50.050838 29555 tablet_service.cc:1514] Processing DeleteTablet for tablet a6ace20ae1d4481798f0a1bd4100a228 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:36398
I20240502 14:14:50.050951 29155 tablet_service.cc:1514] Processing DeleteTablet for tablet a6ace20ae1d4481798f0a1bd4100a228 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:55202
I20240502 14:14:50.065853 29800 tablet_replica.cc:330] T a6ace20ae1d4481798f0a1bd4100a228 P 4092902b3e674857a0147f1b4f2cdb86: stopping tablet replica
I20240502 14:14:50.067533 29800 raft_consensus.cc:2229] T a6ace20ae1d4481798f0a1bd4100a228 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:50.068218 29800 raft_consensus.cc:2258] T a6ace20ae1d4481798f0a1bd4100a228 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:50.069849 29801 tablet_replica.cc:330] T a6ace20ae1d4481798f0a1bd4100a228 P 1cb67158133e4648a6eceba32d4dbba3: stopping tablet replica
I20240502 14:14:50.071988 29801 raft_consensus.cc:2229] T a6ace20ae1d4481798f0a1bd4100a228 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:50.073729 29800 ts_tablet_manager.cc:1875] T a6ace20ae1d4481798f0a1bd4100a228 P 4092902b3e674857a0147f1b4f2cdb86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:50.074281 29801 raft_consensus.cc:2258] T a6ace20ae1d4481798f0a1bd4100a228 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:50.079890 29800 ts_tablet_manager.cc:1888] T a6ace20ae1d4481798f0a1bd4100a228 P 4092902b3e674857a0147f1b4f2cdb86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:50.080075 29800 log.cc:1196] T a6ace20ae1d4481798f0a1bd4100a228 P 4092902b3e674857a0147f1b4f2cdb86: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/wal/wals/a6ace20ae1d4481798f0a1bd4100a228
I20240502 14:14:50.080662 29800 ts_tablet_manager.cc:1909] T a6ace20ae1d4481798f0a1bd4100a228 P 4092902b3e674857a0147f1b4f2cdb86: Deleting consensus metadata
I20240502 14:14:50.082093 29799 tablet_replica.cc:330] T a6ace20ae1d4481798f0a1bd4100a228 P c1100e16b3b9495ca61d4f95e89b6a89: stopping tablet replica
I20240502 14:14:50.082351 29799 raft_consensus.cc:2229] T a6ace20ae1d4481798f0a1bd4100a228 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:50.082492 26272 catalog_manager.cc:4830] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097): tablet a6ace20ae1d4481798f0a1bd4100a228 (table rebalancer_dynamic_table_new_name [id=95cd72c4e03e430b8b16a85239e3d9c8]) successfully deleted
I20240502 14:14:50.082554 29801 ts_tablet_manager.cc:1875] T a6ace20ae1d4481798f0a1bd4100a228 P 1cb67158133e4648a6eceba32d4dbba3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:50.082602 29799 raft_consensus.cc:2258] T a6ace20ae1d4481798f0a1bd4100a228 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:50.084997 29799 ts_tablet_manager.cc:1875] T a6ace20ae1d4481798f0a1bd4100a228 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:50.089298 29643 raft_consensus.cc:482] T abdd0f01d8d5453793c7e9461af10516 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:50.089490 29643 raft_consensus.cc:504] T abdd0f01d8d5453793c7e9461af10516 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:50.090277 29643 leader_election.cc:288] T abdd0f01d8d5453793c7e9461af10516 P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:14:50.092897 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdd0f01d8d5453793c7e9461af10516" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:14:50.093173 29443 raft_consensus.cc:2454] T abdd0f01d8d5453793c7e9461af10516 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 0.
I20240502 14:14:50.100780 29801 ts_tablet_manager.cc:1888] T a6ace20ae1d4481798f0a1bd4100a228 P 1cb67158133e4648a6eceba32d4dbba3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:50.104102 29801 log.cc:1196] T a6ace20ae1d4481798f0a1bd4100a228 P 1cb67158133e4648a6eceba32d4dbba3: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal/wals/a6ace20ae1d4481798f0a1bd4100a228
I20240502 14:14:50.105675 29801 ts_tablet_manager.cc:1909] T a6ace20ae1d4481798f0a1bd4100a228 P 1cb67158133e4648a6eceba32d4dbba3: Deleting consensus metadata
I20240502 14:14:50.108692 29245 leader_election.cc:302] T abdd0f01d8d5453793c7e9461af10516 P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:14:50.109162 29643 raft_consensus.cc:2781] T abdd0f01d8d5453793c7e9461af10516 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:50.109329 29643 raft_consensus.cc:482] T abdd0f01d8d5453793c7e9461af10516 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:50.109454 29643 raft_consensus.cc:3037] T abdd0f01d8d5453793c7e9461af10516 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:50.112594 29643 raft_consensus.cc:504] T abdd0f01d8d5453793c7e9461af10516 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:50.113076 29799 ts_tablet_manager.cc:1888] T a6ace20ae1d4481798f0a1bd4100a228 P c1100e16b3b9495ca61d4f95e89b6a89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:50.113209 29799 log.cc:1196] T a6ace20ae1d4481798f0a1bd4100a228 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal/wals/a6ace20ae1d4481798f0a1bd4100a228
I20240502 14:14:50.113492 29643 leader_election.cc:288] T abdd0f01d8d5453793c7e9461af10516 P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 election: Requested vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:14:50.113760 29799 ts_tablet_manager.cc:1909] T a6ace20ae1d4481798f0a1bd4100a228 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting consensus metadata
I20240502 14:14:50.114928 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdd0f01d8d5453793c7e9461af10516" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
I20240502 14:14:50.115165 29443 raft_consensus.cc:3037] T abdd0f01d8d5453793c7e9461af10516 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:50.116590 26271 catalog_manager.cc:4830] TS 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): tablet a6ace20ae1d4481798f0a1bd4100a228 (table rebalancer_dynamic_table_new_name [id=95cd72c4e03e430b8b16a85239e3d9c8]) successfully deleted
I20240502 14:14:50.117220 26271 catalog_manager.cc:4830] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): tablet a6ace20ae1d4481798f0a1bd4100a228 (table rebalancer_dynamic_table_new_name [id=95cd72c4e03e430b8b16a85239e3d9c8]) successfully deleted
I20240502 14:14:50.118137 29443 raft_consensus.cc:2454] T abdd0f01d8d5453793c7e9461af10516 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 1.
I20240502 14:14:50.120268 29694 raft_consensus.cc:482] T abdd0f01d8d5453793c7e9461af10516 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:50.120481 29694 raft_consensus.cc:504] T abdd0f01d8d5453793c7e9461af10516 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:50.121207 29694 leader_election.cc:288] T abdd0f01d8d5453793c7e9461af10516 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:14:50.122450 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdd0f01d8d5453793c7e9461af10516" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:14:50.122785 29309 raft_consensus.cc:2379] T abdd0f01d8d5453793c7e9461af10516 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4092902b3e674857a0147f1b4f2cdb86 in current term 1: Already voted for candidate c1100e16b3b9495ca61d4f95e89b6a89 in this term.
I20240502 14:14:50.123215 29442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdd0f01d8d5453793c7e9461af10516" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:14:50.126956 29511 leader_election.cc:302] T abdd0f01d8d5453793c7e9461af10516 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:14:50.127534 29694 raft_consensus.cc:3037] T abdd0f01d8d5453793c7e9461af10516 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:50.127498 29575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdd0f01d8d5453793c7e9461af10516" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86" is_pre_election: true
I20240502 14:14:50.127454 29574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdd0f01d8d5453793c7e9461af10516" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86"
I20240502 14:14:50.128542 29245 leader_election.cc:302] T abdd0f01d8d5453793c7e9461af10516 P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:14:50.128937 29643 raft_consensus.cc:2781] T abdd0f01d8d5453793c7e9461af10516 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:50.130573 29694 raft_consensus.cc:2726] T abdd0f01d8d5453793c7e9461af10516 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:14:50.130940 29575 raft_consensus.cc:2454] T abdd0f01d8d5453793c7e9461af10516 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 1.
I20240502 14:14:50.132345 29643 raft_consensus.cc:686] T abdd0f01d8d5453793c7e9461af10516 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 LEADER]: Becoming Leader. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Running, Role: LEADER
I20240502 14:14:50.133085 29643 consensus_queue.cc:237] T abdd0f01d8d5453793c7e9461af10516 P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:50.138082 26278 catalog_manager.cc:5484] T abdd0f01d8d5453793c7e9461af10516 P c1100e16b3b9495ca61d4f95e89b6a89 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198). New cstate: current_term: 1 leader_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:14:50.153189 26278 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:14:50.164885 26537 tablet_service.cc:1467] Processing CreateTablet for tablet ca7bf077ff4945089c6cc9cdc1d1b5f9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=fddfe01d8d2b414896e3001ff997b4d2]), partition=
I20240502 14:14:50.165184 29289 tablet_service.cc:1467] Processing CreateTablet for tablet ca7bf077ff4945089c6cc9cdc1d1b5f9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=fddfe01d8d2b414896e3001ff997b4d2]), partition=
I20240502 14:14:50.165431 26537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca7bf077ff4945089c6cc9cdc1d1b5f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:50.165602 29289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca7bf077ff4945089c6cc9cdc1d1b5f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:50.168589 29555 tablet_service.cc:1467] Processing CreateTablet for tablet ca7bf077ff4945089c6cc9cdc1d1b5f9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=fddfe01d8d2b414896e3001ff997b4d2]), partition=
I20240502 14:14:50.169030 29555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca7bf077ff4945089c6cc9cdc1d1b5f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:50.172564 29004 raft_consensus.cc:482] T f8923ed1ce94441a8eb303b6f7dd3acb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:50.172775 29004 raft_consensus.cc:504] T f8923ed1ce94441a8eb303b6f7dd3acb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:50.173779 29004 leader_election.cc:288] T f8923ed1ce94441a8eb303b6f7dd3acb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:14:50.186853 29818 tablet_bootstrap.cc:492] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:50.191532 29818 tablet_bootstrap.cc:654] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:50.193019 29639 tablet_bootstrap.cc:492] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:14:50.195233 29639 tablet_bootstrap.cc:654] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:50.198501 29639 tablet_bootstrap.cc:492] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:14:50.198849 29639 ts_tablet_manager.cc:1367] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240502 14:14:50.200968 29639 raft_consensus.cc:348] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:50.201372 29818 tablet_bootstrap.cc:492] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:50.201723 29639 raft_consensus.cc:374] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:50.201848 29639 raft_consensus.cc:729] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:14:50.202009 29818 ts_tablet_manager.cc:1367] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.004s
I20240502 14:14:50.202236 29639 consensus_queue.cc:260] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:50.203354 29639 ts_tablet_manager.cc:1398] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 14:14:50.203891 29777 tablet_bootstrap.cc:492] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:14:50.206527 29777 tablet_bootstrap.cc:654] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:50.209671 29818 raft_consensus.cc:348] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:50.210065 29818 raft_consensus.cc:374] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:50.211539 29694 raft_consensus.cc:482] T f8923ed1ce94441a8eb303b6f7dd3acb P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:50.211750 29694 raft_consensus.cc:504] T f8923ed1ce94441a8eb303b6f7dd3acb P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:50.212285 29818 raft_consensus.cc:729] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:50.212635 29694 leader_election.cc:288] T f8923ed1ce94441a8eb303b6f7dd3acb P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:14:50.213197 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8923ed1ce94441a8eb303b6f7dd3acb" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:14:50.213486 29443 raft_consensus.cc:2454] T f8923ed1ce94441a8eb303b6f7dd3acb P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:14:50.213853 26687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8923ed1ce94441a8eb303b6f7dd3acb" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:14:50.214087 26687 raft_consensus.cc:2454] T f8923ed1ce94441a8eb303b6f7dd3acb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:14:50.214591 29511 leader_election.cc:302] T f8923ed1ce94441a8eb303b6f7dd3acb P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, 68a3c2b17ed94e06aba66ed4178132ab; no voters:
I20240502 14:14:50.215305 29694 raft_consensus.cc:2781] T f8923ed1ce94441a8eb303b6f7dd3acb P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:50.215411 29694 raft_consensus.cc:482] T f8923ed1ce94441a8eb303b6f7dd3acb P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:50.215528 29694 raft_consensus.cc:3037] T f8923ed1ce94441a8eb303b6f7dd3acb P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:50.218169 29694 raft_consensus.cc:504] T f8923ed1ce94441a8eb303b6f7dd3acb P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:50.218988 29694 leader_election.cc:288] T f8923ed1ce94441a8eb303b6f7dd3acb P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Requested vote from peers 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:14:50.219816 26687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8923ed1ce94441a8eb303b6f7dd3acb" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:14:50.220068 26687 raft_consensus.cc:3037] T f8923ed1ce94441a8eb303b6f7dd3acb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:50.220098 29818 consensus_queue.cc:260] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:50.221199 29575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8923ed1ce94441a8eb303b6f7dd3acb" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86" is_pre_election: true
I20240502 14:14:50.221603 29575 raft_consensus.cc:2379] T f8923ed1ce94441a8eb303b6f7dd3acb P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in current term 1: Already voted for candidate 4092902b3e674857a0147f1b4f2cdb86 in this term.
I20240502 14:14:50.222987 29818 ts_tablet_manager.cc:1398] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.021s user 0.001s sys 0.001s
I20240502 14:14:50.229966 29777 tablet_bootstrap.cc:492] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:14:50.230013 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8923ed1ce94441a8eb303b6f7dd3acb" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
I20240502 14:14:50.230221 29777 ts_tablet_manager.cc:1367] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.002s
I20240502 14:14:50.230273 29443 raft_consensus.cc:3037] T f8923ed1ce94441a8eb303b6f7dd3acb P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:50.231308 29777 raft_consensus.cc:348] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:50.231632 29777 raft_consensus.cc:374] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:50.231721 29777 raft_consensus.cc:729] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:14:50.232107 29777 consensus_queue.cc:260] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:50.235054 29777 ts_tablet_manager.cc:1398] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 14:14:50.235637 29442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8923ed1ce94441a8eb303b6f7dd3acb" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:14:50.236472 26625 leader_election.cc:302] T f8923ed1ce94441a8eb303b6f7dd3acb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters: 4092902b3e674857a0147f1b4f2cdb86, 68a3c2b17ed94e06aba66ed4178132ab
I20240502 14:14:50.245903 26687 raft_consensus.cc:2454] T f8923ed1ce94441a8eb303b6f7dd3acb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:14:50.246268 29443 raft_consensus.cc:2454] T f8923ed1ce94441a8eb303b6f7dd3acb P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:14:50.246902 29511 leader_election.cc:302] T f8923ed1ce94441a8eb303b6f7dd3acb P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, 68a3c2b17ed94e06aba66ed4178132ab; no voters:
I20240502 14:14:50.247239 29694 raft_consensus.cc:2781] T f8923ed1ce94441a8eb303b6f7dd3acb P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:50.247385 29004 raft_consensus.cc:2726] T f8923ed1ce94441a8eb303b6f7dd3acb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:14:50.247412 29694 raft_consensus.cc:686] T f8923ed1ce94441a8eb303b6f7dd3acb P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Becoming Leader. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Running, Role: LEADER
I20240502 14:14:50.248172 29694 consensus_queue.cc:237] T f8923ed1ce94441a8eb303b6f7dd3acb P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:50.265554 26284 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240502 14:14:50.265861 26284 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240502 14:14:50.273494 26284 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet df0c1182765041a7ad4f28ef6aff35f0
I20240502 14:14:50.274528 26406 tablet_service.cc:1514] Processing DeleteTablet for tablet df0c1182765041a7ad4f28ef6aff35f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:37422
I20240502 14:14:50.275005 29782 tablet_replica.cc:330] T df0c1182765041a7ad4f28ef6aff35f0 P 89af7f423cf44717a08d6b142f0ed49d: stopping tablet replica
I20240502 14:14:50.275275 29782 raft_consensus.cc:2229] T df0c1182765041a7ad4f28ef6aff35f0 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:50.275672 29782 raft_consensus.cc:2258] T df0c1182765041a7ad4f28ef6aff35f0 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:50.276806 29782 ts_tablet_manager.cc:1875] T df0c1182765041a7ad4f28ef6aff35f0 P 89af7f423cf44717a08d6b142f0ed49d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:50.280095 26284 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet 6d8b293ae36740d08d64bdeeb197897d
I20240502 14:14:50.280203 26537 tablet_service.cc:1514] Processing DeleteTablet for tablet df0c1182765041a7ad4f28ef6aff35f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:51696
I20240502 14:14:50.280570 29781 tablet_replica.cc:330] T df0c1182765041a7ad4f28ef6aff35f0 P ba187e38a2754d60a12de9ac4856430a: stopping tablet replica
I20240502 14:14:50.280906 29781 raft_consensus.cc:2229] T df0c1182765041a7ad4f28ef6aff35f0 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:50.281283 29781 raft_consensus.cc:2258] T df0c1182765041a7ad4f28ef6aff35f0 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:50.282442 29781 ts_tablet_manager.cc:1875] T df0c1182765041a7ad4f28ef6aff35f0 P ba187e38a2754d60a12de9ac4856430a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:50.284562 26797 tablet_service.cc:1514] Processing DeleteTablet for tablet df0c1182765041a7ad4f28ef6aff35f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:56010
I20240502 14:14:50.284876 29783 tablet_replica.cc:330] T df0c1182765041a7ad4f28ef6aff35f0 P e4cfb8fc5a474177af6848a6efa13a27: stopping tablet replica
I20240502 14:14:50.285095 29783 raft_consensus.cc:2229] T df0c1182765041a7ad4f28ef6aff35f0 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:50.285377 29783 raft_consensus.cc:2258] T df0c1182765041a7ad4f28ef6aff35f0 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:50.286191 29783 ts_tablet_manager.cc:1875] T df0c1182765041a7ad4f28ef6aff35f0 P e4cfb8fc5a474177af6848a6efa13a27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:50.288635 26669 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d8b293ae36740d08d64bdeeb197897d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:34150
I20240502 14:14:50.289014 29784 tablet_replica.cc:330] T 6d8b293ae36740d08d64bdeeb197897d P 69ab78b2a1b74d2bbf2cdeafa0a4af23: stopping tablet replica
I20240502 14:14:50.289038 26406 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d8b293ae36740d08d64bdeeb197897d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:37422
I20240502 14:14:50.289237 29784 raft_consensus.cc:2229] T 6d8b293ae36740d08d64bdeeb197897d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:50.289743 29784 raft_consensus.cc:2258] T 6d8b293ae36740d08d64bdeeb197897d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:50.290203 26284 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet 78954e9f7e3147e288eb0cbf6396d345
I20240502 14:14:50.290474 26797 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d8b293ae36740d08d64bdeeb197897d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:56010
I20240502 14:14:50.290776 29784 ts_tablet_manager.cc:1875] T 6d8b293ae36740d08d64bdeeb197897d P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:50.291077 26797 tablet_service.cc:1514] Processing DeleteTablet for tablet 78954e9f7e3147e288eb0cbf6396d345 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:56010
I20240502 14:14:50.292001 26406 tablet_service.cc:1514] Processing DeleteTablet for tablet 78954e9f7e3147e288eb0cbf6396d345 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:37422
I20240502 14:14:50.293212 26538 tablet_service.cc:1514] Processing DeleteTablet for tablet 78954e9f7e3147e288eb0cbf6396d345 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:51696
I20240502 14:14:50.297233 29783 ts_tablet_manager.cc:1888] T df0c1182765041a7ad4f28ef6aff35f0 P e4cfb8fc5a474177af6848a6efa13a27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:50.297456 29783 log.cc:1196] T df0c1182765041a7ad4f28ef6aff35f0 P e4cfb8fc5a474177af6848a6efa13a27: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/wals/df0c1182765041a7ad4f28ef6aff35f0
I20240502 14:14:50.298007 29783 ts_tablet_manager.cc:1909] T df0c1182765041a7ad4f28ef6aff35f0 P e4cfb8fc5a474177af6848a6efa13a27: Deleting consensus metadata
I20240502 14:14:50.299296 26272 catalog_manager.cc:4830] TS e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): tablet df0c1182765041a7ad4f28ef6aff35f0 (table rebalance_test_table_6 [id=78d3d1a6e46640a0a44b6b531584cbe1]) successfully deleted
I20240502 14:14:50.299768 29783 tablet_replica.cc:330] T 6d8b293ae36740d08d64bdeeb197897d P e4cfb8fc5a474177af6848a6efa13a27: stopping tablet replica
I20240502 14:14:50.300148 29783 raft_consensus.cc:2229] T 6d8b293ae36740d08d64bdeeb197897d P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:50.300491 29783 raft_consensus.cc:2258] T 6d8b293ae36740d08d64bdeeb197897d P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:50.301920 29781 ts_tablet_manager.cc:1888] T df0c1182765041a7ad4f28ef6aff35f0 P ba187e38a2754d60a12de9ac4856430a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:50.302084 29781 log.cc:1196] T df0c1182765041a7ad4f28ef6aff35f0 P ba187e38a2754d60a12de9ac4856430a: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/df0c1182765041a7ad4f28ef6aff35f0
I20240502 14:14:50.302325 29784 ts_tablet_manager.cc:1888] T 6d8b293ae36740d08d64bdeeb197897d P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:50.302423 29783 ts_tablet_manager.cc:1875] T 6d8b293ae36740d08d64bdeeb197897d P e4cfb8fc5a474177af6848a6efa13a27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:50.302513 29784 log.cc:1196] T 6d8b293ae36740d08d64bdeeb197897d P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/wals/6d8b293ae36740d08d64bdeeb197897d
I20240502 14:14:50.302861 29782 ts_tablet_manager.cc:1888] T df0c1182765041a7ad4f28ef6aff35f0 P 89af7f423cf44717a08d6b142f0ed49d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:50.303066 29784 ts_tablet_manager.cc:1909] T 6d8b293ae36740d08d64bdeeb197897d P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting consensus metadata
I20240502 14:14:50.304991 26272 catalog_manager.cc:4830] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): tablet 6d8b293ae36740d08d64bdeeb197897d (table rebalance_test_table_6 [id=78d3d1a6e46640a0a44b6b531584cbe1]) successfully deleted
I20240502 14:14:50.306519 29781 ts_tablet_manager.cc:1909] T df0c1182765041a7ad4f28ef6aff35f0 P ba187e38a2754d60a12de9ac4856430a: Deleting consensus metadata
I20240502 14:14:50.306792 29782 log.cc:1196] T df0c1182765041a7ad4f28ef6aff35f0 P 89af7f423cf44717a08d6b142f0ed49d: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/df0c1182765041a7ad4f28ef6aff35f0
I20240502 14:14:50.307425 29782 ts_tablet_manager.cc:1909] T df0c1182765041a7ad4f28ef6aff35f0 P 89af7f423cf44717a08d6b142f0ed49d: Deleting consensus metadata
I20240502 14:14:50.308884 26271 catalog_manager.cc:4830] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): tablet df0c1182765041a7ad4f28ef6aff35f0 (table rebalance_test_table_6 [id=78d3d1a6e46640a0a44b6b531584cbe1]) successfully deleted
I20240502 14:14:50.312074 26271 catalog_manager.cc:4830] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): tablet df0c1182765041a7ad4f28ef6aff35f0 (table rebalance_test_table_6 [id=78d3d1a6e46640a0a44b6b531584cbe1]) successfully deleted
I20240502 14:14:50.313082 29782 tablet_replica.cc:330] T 6d8b293ae36740d08d64bdeeb197897d P 89af7f423cf44717a08d6b142f0ed49d: stopping tablet replica
I20240502 14:14:50.313956 29782 raft_consensus.cc:2229] T 6d8b293ae36740d08d64bdeeb197897d P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:50.313421 29781 tablet_replica.cc:330] T 78954e9f7e3147e288eb0cbf6396d345 P ba187e38a2754d60a12de9ac4856430a: stopping tablet replica
I20240502 14:14:50.314538 29781 raft_consensus.cc:2229] T 78954e9f7e3147e288eb0cbf6396d345 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:50.315039 29782 raft_consensus.cc:2258] T 6d8b293ae36740d08d64bdeeb197897d P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:50.316008 29781 raft_consensus.cc:2258] T 78954e9f7e3147e288eb0cbf6396d345 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:50.317569 29783 ts_tablet_manager.cc:1888] T 6d8b293ae36740d08d64bdeeb197897d P e4cfb8fc5a474177af6848a6efa13a27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:50.317597 29782 ts_tablet_manager.cc:1875] T 6d8b293ae36740d08d64bdeeb197897d P 89af7f423cf44717a08d6b142f0ed49d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:50.318017 29783 log.cc:1196] T 6d8b293ae36740d08d64bdeeb197897d P e4cfb8fc5a474177af6848a6efa13a27: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/wals/6d8b293ae36740d08d64bdeeb197897d
I20240502 14:14:50.318565 29783 ts_tablet_manager.cc:1909] T 6d8b293ae36740d08d64bdeeb197897d P e4cfb8fc5a474177af6848a6efa13a27: Deleting consensus metadata
I20240502 14:14:50.319818 26272 catalog_manager.cc:4830] TS e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): tablet 6d8b293ae36740d08d64bdeeb197897d (table rebalance_test_table_6 [id=78d3d1a6e46640a0a44b6b531584cbe1]) successfully deleted
I20240502 14:14:50.320297 29783 tablet_replica.cc:330] T 78954e9f7e3147e288eb0cbf6396d345 P e4cfb8fc5a474177af6848a6efa13a27: stopping tablet replica
I20240502 14:14:50.320125 26284 catalog_manager.cc:5484] T f8923ed1ce94441a8eb303b6f7dd3acb P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200). New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:14:50.320618 29783 raft_consensus.cc:2229] T 78954e9f7e3147e288eb0cbf6396d345 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:50.320922 29783 raft_consensus.cc:2258] T 78954e9f7e3147e288eb0cbf6396d345 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:50.321722 29783 ts_tablet_manager.cc:1875] T 78954e9f7e3147e288eb0cbf6396d345 P e4cfb8fc5a474177af6848a6efa13a27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:50.321800 29781 ts_tablet_manager.cc:1875] T 78954e9f7e3147e288eb0cbf6396d345 P ba187e38a2754d60a12de9ac4856430a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:50.328922 29783 ts_tablet_manager.cc:1888] T 78954e9f7e3147e288eb0cbf6396d345 P e4cfb8fc5a474177af6848a6efa13a27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:50.329154 29783 log.cc:1196] T 78954e9f7e3147e288eb0cbf6396d345 P e4cfb8fc5a474177af6848a6efa13a27: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/wals/78954e9f7e3147e288eb0cbf6396d345
I20240502 14:14:50.329706 29783 ts_tablet_manager.cc:1909] T 78954e9f7e3147e288eb0cbf6396d345 P e4cfb8fc5a474177af6848a6efa13a27: Deleting consensus metadata
I20240502 14:14:50.330962 26272 catalog_manager.cc:4830] TS e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): tablet 78954e9f7e3147e288eb0cbf6396d345 (table rebalance_test_table_6 [id=78d3d1a6e46640a0a44b6b531584cbe1]) successfully deleted
I20240502 14:14:50.333396 29781 ts_tablet_manager.cc:1888] T 78954e9f7e3147e288eb0cbf6396d345 P ba187e38a2754d60a12de9ac4856430a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:50.333566 29781 log.cc:1196] T 78954e9f7e3147e288eb0cbf6396d345 P ba187e38a2754d60a12de9ac4856430a: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/78954e9f7e3147e288eb0cbf6396d345
I20240502 14:14:50.334229 29781 ts_tablet_manager.cc:1909] T 78954e9f7e3147e288eb0cbf6396d345 P ba187e38a2754d60a12de9ac4856430a: Deleting consensus metadata
I20240502 14:14:50.335426 26271 catalog_manager.cc:4830] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): tablet 78954e9f7e3147e288eb0cbf6396d345 (table rebalance_test_table_6 [id=78d3d1a6e46640a0a44b6b531584cbe1]) successfully deleted
I20240502 14:14:50.335888 29782 ts_tablet_manager.cc:1888] T 6d8b293ae36740d08d64bdeeb197897d P 89af7f423cf44717a08d6b142f0ed49d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:50.336048 29782 log.cc:1196] T 6d8b293ae36740d08d64bdeeb197897d P 89af7f423cf44717a08d6b142f0ed49d: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/6d8b293ae36740d08d64bdeeb197897d
I20240502 14:14:50.336621 29782 ts_tablet_manager.cc:1909] T 6d8b293ae36740d08d64bdeeb197897d P 89af7f423cf44717a08d6b142f0ed49d: Deleting consensus metadata
I20240502 14:14:50.337695 29782 tablet_replica.cc:330] T 78954e9f7e3147e288eb0cbf6396d345 P 89af7f423cf44717a08d6b142f0ed49d: stopping tablet replica
I20240502 14:14:50.337925 29782 raft_consensus.cc:2229] T 78954e9f7e3147e288eb0cbf6396d345 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:50.338115 26271 catalog_manager.cc:4830] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): tablet 6d8b293ae36740d08d64bdeeb197897d (table rebalance_test_table_6 [id=78d3d1a6e46640a0a44b6b531584cbe1]) successfully deleted
I20240502 14:14:50.338254 29782 raft_consensus.cc:2258] T 78954e9f7e3147e288eb0cbf6396d345 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:50.339082 29782 ts_tablet_manager.cc:1875] T 78954e9f7e3147e288eb0cbf6396d345 P 89af7f423cf44717a08d6b142f0ed49d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:50.346243 29782 ts_tablet_manager.cc:1888] T 78954e9f7e3147e288eb0cbf6396d345 P 89af7f423cf44717a08d6b142f0ed49d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:50.346426 29782 log.cc:1196] T 78954e9f7e3147e288eb0cbf6396d345 P 89af7f423cf44717a08d6b142f0ed49d: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/78954e9f7e3147e288eb0cbf6396d345
I20240502 14:14:50.346868 29782 ts_tablet_manager.cc:1909] T 78954e9f7e3147e288eb0cbf6396d345 P 89af7f423cf44717a08d6b142f0ed49d: Deleting consensus metadata
I20240502 14:14:50.348546 26271 catalog_manager.cc:4830] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): tablet 78954e9f7e3147e288eb0cbf6396d345 (table rebalance_test_table_6 [id=78d3d1a6e46640a0a44b6b531584cbe1]) successfully deleted
I20240502 14:14:50.390704 29643 raft_consensus.cc:482] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:50.391037 29643 raft_consensus.cc:504] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:50.391754 29643 leader_election.cc:288] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:14:50.392236 26557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca7bf077ff4945089c6cc9cdc1d1b5f9" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:50.392359 29575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca7bf077ff4945089c6cc9cdc1d1b5f9" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86" is_pre_election: true
I20240502 14:14:50.392495 26557 raft_consensus.cc:2454] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 0.
I20240502 14:14:50.392632 29575 raft_consensus.cc:2454] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 0.
I20240502 14:14:50.393081 29245 leader_election.cc:302] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:14:50.393419 29643 raft_consensus.cc:2781] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:50.393532 29643 raft_consensus.cc:482] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:50.393642 29643 raft_consensus.cc:3037] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:50.396777 29643 raft_consensus.cc:504] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:50.398224 26557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca7bf077ff4945089c6cc9cdc1d1b5f9" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:14:50.398322 29575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca7bf077ff4945089c6cc9cdc1d1b5f9" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86"
I20240502 14:14:50.398464 26557 raft_consensus.cc:3037] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:50.398527 29575 raft_consensus.cc:3037] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:50.401530 26557 raft_consensus.cc:2454] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 1.
I20240502 14:14:50.401531 29575 raft_consensus.cc:2454] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 1.
I20240502 14:14:50.402081 29244 leader_election.cc:302] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:14:50.402523 29805 raft_consensus.cc:2781] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:50.403060 29805 raft_consensus.cc:686] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 LEADER]: Becoming Leader. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Running, Role: LEADER
I20240502 14:14:50.403482 29805 consensus_queue.cc:237] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:50.404619 29643 leader_election.cc:288] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 election: Requested vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:14:50.407632 26284 catalog_manager.cc:5484] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P c1100e16b3b9495ca61d4f95e89b6a89 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198). New cstate: current_term: 1 leader_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:14:50.426115 29694 consensus_queue.cc:1035] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:50.457140 29704 consensus_queue.cc:1035] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:50.551620 29805 consensus_queue.cc:1035] T abdd0f01d8d5453793c7e9461af10516 P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:50.556941 26283 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:14:50.565673 26558 raft_consensus.cc:1261] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer c1100e16b3b9495ca61d4f95e89b6a89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:50.566078 29575 raft_consensus.cc:1261] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Refusing update from remote peer c1100e16b3b9495ca61d4f95e89b6a89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:50.564620 29643 consensus_queue.cc:1035] T abdd0f01d8d5453793c7e9461af10516 P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240502 14:14:50.567000 29840 consensus_queue.cc:1035] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:50.570605 29643 consensus_queue.cc:1035] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:50.580135 26277 catalog_manager.cc:5576] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025) does not have the latest schema for tablet ca7bf077ff4945089c6cc9cdc1d1b5f9 (table rebalancer_dynamic_table [id=fddfe01d8d2b414896e3001ff997b4d2]). Expected version 1 got 0
I20240502 14:14:50.580935 26283 catalog_manager.cc:5576] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097) does not have the latest schema for tablet ca7bf077ff4945089c6cc9cdc1d1b5f9 (table rebalancer_dynamic_table [id=fddfe01d8d2b414896e3001ff997b4d2]). Expected version 1 got 0
I20240502 14:14:50.580982 29873 tablet.cc:1722] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:50.582139 29873 tablet.cc:1620] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P c1100e16b3b9495ca61d4f95e89b6a89: MemRowSet was empty: no flush needed.
I20240502 14:14:50.583042 29867 mvcc.cc:205] Tried to move back new op lower bound from 7023244454151049216 to 7023244453495922688. Current Snapshot: MvccSnapshot[applied={T|T < 7023244454151049216}]
I20240502 14:14:50.589274 29874 tablet.cc:1722] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P ba187e38a2754d60a12de9ac4856430a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:50.590075 29874 tablet.cc:1620] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P ba187e38a2754d60a12de9ac4856430a: MemRowSet was empty: no flush needed.
I20240502 14:14:50.597812 26277 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=fddfe01d8d2b414896e3001ff997b4d2] alter complete (version 1)
I20240502 14:14:50.599285 26283 catalog_manager.cc:5576] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025) does not have the latest schema for tablet ca7bf077ff4945089c6cc9cdc1d1b5f9 (table rebalancer_dynamic_table [id=fddfe01d8d2b414896e3001ff997b4d2]). Expected version 1 got 0
I20240502 14:14:50.618849 26283 catalog_manager.cc:5576] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097) does not have the latest schema for tablet ca7bf077ff4945089c6cc9cdc1d1b5f9 (table rebalancer_dynamic_table [id=fddfe01d8d2b414896e3001ff997b4d2]). Expected version 1 got 0
I20240502 14:14:50.616169 29876 tablet.cc:1722] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:50.625416 29155 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:14:50.623147 29876 tablet.cc:1620] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P 4092902b3e674857a0147f1b4f2cdb86: MemRowSet was empty: no flush needed.
I20240502 14:14:50.677094 26406 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:14:50.678202 26537 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:14:50.698275 29555 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:50.701956 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:14:50.702442 26283 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:14:50.710464 29421 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:50.727480 29873 tablet.cc:1722] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:50.728019 29873 tablet.cc:1620] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P c1100e16b3b9495ca61d4f95e89b6a89: MemRowSet was empty: no flush needed.
I20240502 14:14:50.733633 29876 tablet.cc:1722] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:50.734210 29876 tablet.cc:1620] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P 4092902b3e674857a0147f1b4f2cdb86: MemRowSet was empty: no flush needed.
I20240502 14:14:50.737171 29289 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:50.738876 29874 tablet.cc:1722] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P ba187e38a2754d60a12de9ac4856430a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:50.739472 29874 tablet.cc:1620] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P ba187e38a2754d60a12de9ac4856430a: MemRowSet was empty: no flush needed.
I20240502 14:14:50.744772 29694 consensus_queue.cc:1035] T f8923ed1ce94441a8eb303b6f7dd3acb P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:50.752060 26283 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=fddfe01d8d2b414896e3001ff997b4d2] alter complete (version 2)
I20240502 14:14:50.762326 26797 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:14:50.794484 26277 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240502 14:14:50.794800 26277 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240502 14:14:50.800459 26277 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet 11f5b70d258c445a8d3b09bd5cd46965
I20240502 14:14:50.801493 26537 tablet_service.cc:1514] Processing DeleteTablet for tablet 11f5b70d258c445a8d3b09bd5cd46965 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:51696
I20240502 14:14:50.801458 29694 consensus_queue.cc:1035] T f8923ed1ce94441a8eb303b6f7dd3acb P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:50.801831 29781 tablet_replica.cc:330] T 11f5b70d258c445a8d3b09bd5cd46965 P ba187e38a2754d60a12de9ac4856430a: stopping tablet replica
I20240502 14:14:50.802078 29781 raft_consensus.cc:2229] T 11f5b70d258c445a8d3b09bd5cd46965 P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:50.802469 29781 raft_consensus.cc:2258] T 11f5b70d258c445a8d3b09bd5cd46965 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:50.803315 29781 ts_tablet_manager.cc:1875] T 11f5b70d258c445a8d3b09bd5cd46965 P ba187e38a2754d60a12de9ac4856430a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:50.807786 26406 tablet_service.cc:1514] Processing DeleteTablet for tablet 11f5b70d258c445a8d3b09bd5cd46965 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:37422
I20240502 14:14:50.808176 29782 tablet_replica.cc:330] T 11f5b70d258c445a8d3b09bd5cd46965 P 89af7f423cf44717a08d6b142f0ed49d: stopping tablet replica
I20240502 14:14:50.808463 29782 raft_consensus.cc:2229] T 11f5b70d258c445a8d3b09bd5cd46965 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:50.808750 29782 raft_consensus.cc:2258] T 11f5b70d258c445a8d3b09bd5cd46965 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:50.808732 26797 tablet_service.cc:1514] Processing DeleteTablet for tablet 11f5b70d258c445a8d3b09bd5cd46965 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:56010
I20240502 14:14:50.808773 26277 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet 56dddccc99524329bf2992a14c6235c7
I20240502 14:14:50.809078 29783 tablet_replica.cc:330] T 11f5b70d258c445a8d3b09bd5cd46965 P e4cfb8fc5a474177af6848a6efa13a27: stopping tablet replica
I20240502 14:14:50.809314 29783 raft_consensus.cc:2229] T 11f5b70d258c445a8d3b09bd5cd46965 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:50.809557 29783 raft_consensus.cc:2258] T 11f5b70d258c445a8d3b09bd5cd46965 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:50.809705 29782 ts_tablet_manager.cc:1875] T 11f5b70d258c445a8d3b09bd5cd46965 P 89af7f423cf44717a08d6b142f0ed49d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:50.809783 26669 tablet_service.cc:1514] Processing DeleteTablet for tablet 56dddccc99524329bf2992a14c6235c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:34150
I20240502 14:14:50.810508 29783 ts_tablet_manager.cc:1875] T 11f5b70d258c445a8d3b09bd5cd46965 P e4cfb8fc5a474177af6848a6efa13a27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:50.811673 26406 tablet_service.cc:1514] Processing DeleteTablet for tablet 56dddccc99524329bf2992a14c6235c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:37422
I20240502 14:14:50.812582 26537 tablet_service.cc:1514] Processing DeleteTablet for tablet 56dddccc99524329bf2992a14c6235c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:51696
I20240502 14:14:50.815353 26277 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet 64af9832dbc0464d8eea5f434172ee1f
I20240502 14:14:50.816232 26797 tablet_service.cc:1514] Processing DeleteTablet for tablet 64af9832dbc0464d8eea5f434172ee1f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:56010
I20240502 14:14:50.817291 26668 tablet_service.cc:1514] Processing DeleteTablet for tablet 64af9832dbc0464d8eea5f434172ee1f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:34150
I20240502 14:14:50.819478 26406 tablet_service.cc:1514] Processing DeleteTablet for tablet 64af9832dbc0464d8eea5f434172ee1f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:37422
I20240502 14:14:50.819811 29782 ts_tablet_manager.cc:1888] T 11f5b70d258c445a8d3b09bd5cd46965 P 89af7f423cf44717a08d6b142f0ed49d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:50.824697 29782 log.cc:1196] T 11f5b70d258c445a8d3b09bd5cd46965 P 89af7f423cf44717a08d6b142f0ed49d: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/11f5b70d258c445a8d3b09bd5cd46965
I20240502 14:14:50.826130 29782 ts_tablet_manager.cc:1909] T 11f5b70d258c445a8d3b09bd5cd46965 P 89af7f423cf44717a08d6b142f0ed49d: Deleting consensus metadata
I20240502 14:14:50.833818 29782 tablet_replica.cc:330] T 56dddccc99524329bf2992a14c6235c7 P 89af7f423cf44717a08d6b142f0ed49d: stopping tablet replica
I20240502 14:14:50.833999 26271 catalog_manager.cc:4830] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): tablet 11f5b70d258c445a8d3b09bd5cd46965 (table rebalance_test_table_8 [id=a0c39b03bebf4af19fe58980960cc466]) successfully deleted
I20240502 14:14:50.834290 29782 raft_consensus.cc:2229] T 56dddccc99524329bf2992a14c6235c7 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:50.834775 29782 raft_consensus.cc:2258] T 56dddccc99524329bf2992a14c6235c7 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:50.835664 29782 ts_tablet_manager.cc:1875] T 56dddccc99524329bf2992a14c6235c7 P 89af7f423cf44717a08d6b142f0ed49d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:50.836194 29783 ts_tablet_manager.cc:1888] T 11f5b70d258c445a8d3b09bd5cd46965 P e4cfb8fc5a474177af6848a6efa13a27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:50.838522 29783 log.cc:1196] T 11f5b70d258c445a8d3b09bd5cd46965 P e4cfb8fc5a474177af6848a6efa13a27: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/wals/11f5b70d258c445a8d3b09bd5cd46965
I20240502 14:14:50.839205 29783 ts_tablet_manager.cc:1909] T 11f5b70d258c445a8d3b09bd5cd46965 P e4cfb8fc5a474177af6848a6efa13a27: Deleting consensus metadata
I20240502 14:14:50.841325 29781 ts_tablet_manager.cc:1888] T 11f5b70d258c445a8d3b09bd5cd46965 P ba187e38a2754d60a12de9ac4856430a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:50.842007 29781 log.cc:1196] T 11f5b70d258c445a8d3b09bd5cd46965 P ba187e38a2754d60a12de9ac4856430a: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/11f5b70d258c445a8d3b09bd5cd46965
I20240502 14:14:50.842700 29893 tablet_replica.cc:330] T 56dddccc99524329bf2992a14c6235c7 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: stopping tablet replica
I20240502 14:14:50.843827 29893 raft_consensus.cc:2229] T 56dddccc99524329bf2992a14c6235c7 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:50.844275 26272 catalog_manager.cc:4830] TS e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): tablet 11f5b70d258c445a8d3b09bd5cd46965 (table rebalance_test_table_8 [id=a0c39b03bebf4af19fe58980960cc466]) successfully deleted
I20240502 14:14:50.845146 29781 ts_tablet_manager.cc:1909] T 11f5b70d258c445a8d3b09bd5cd46965 P ba187e38a2754d60a12de9ac4856430a: Deleting consensus metadata
I20240502 14:14:50.845657 29893 raft_consensus.cc:2258] T 56dddccc99524329bf2992a14c6235c7 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:50.846019 29783 tablet_replica.cc:330] T 64af9832dbc0464d8eea5f434172ee1f P e4cfb8fc5a474177af6848a6efa13a27: stopping tablet replica
I20240502 14:14:50.848889 29783 raft_consensus.cc:2229] T 64af9832dbc0464d8eea5f434172ee1f P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:50.851374 26271 catalog_manager.cc:4830] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): tablet 11f5b70d258c445a8d3b09bd5cd46965 (table rebalance_test_table_8 [id=a0c39b03bebf4af19fe58980960cc466]) successfully deleted
I20240502 14:14:50.852701 29781 tablet_replica.cc:330] T 56dddccc99524329bf2992a14c6235c7 P ba187e38a2754d60a12de9ac4856430a: stopping tablet replica
I20240502 14:14:50.853288 29781 raft_consensus.cc:2229] T 56dddccc99524329bf2992a14c6235c7 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:50.853804 29781 raft_consensus.cc:2258] T 56dddccc99524329bf2992a14c6235c7 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:50.855110 29781 ts_tablet_manager.cc:1875] T 56dddccc99524329bf2992a14c6235c7 P ba187e38a2754d60a12de9ac4856430a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:50.855130 29783 raft_consensus.cc:2258] T 64af9832dbc0464d8eea5f434172ee1f P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:50.856720 26277 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:14:50.859426 29893 ts_tablet_manager.cc:1875] T 56dddccc99524329bf2992a14c6235c7 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:50.860311 29783 ts_tablet_manager.cc:1875] T 64af9832dbc0464d8eea5f434172ee1f P e4cfb8fc5a474177af6848a6efa13a27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:50.873489 29873 tablet.cc:1722] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:50.873687 29781 ts_tablet_manager.cc:1888] T 56dddccc99524329bf2992a14c6235c7 P ba187e38a2754d60a12de9ac4856430a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:50.873859 29781 log.cc:1196] T 56dddccc99524329bf2992a14c6235c7 P ba187e38a2754d60a12de9ac4856430a: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/56dddccc99524329bf2992a14c6235c7
I20240502 14:14:50.874584 29781 ts_tablet_manager.cc:1909] T 56dddccc99524329bf2992a14c6235c7 P ba187e38a2754d60a12de9ac4856430a: Deleting consensus metadata
I20240502 14:14:50.877269 29874 tablet.cc:1722] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P ba187e38a2754d60a12de9ac4856430a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:50.877424 26271 catalog_manager.cc:4830] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): tablet 56dddccc99524329bf2992a14c6235c7 (table rebalance_test_table_8 [id=a0c39b03bebf4af19fe58980960cc466]) successfully deleted
I20240502 14:14:50.889401 26277 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table_new_name [id=fddfe01d8d2b414896e3001ff997b4d2] alter complete (version 3)
I20240502 14:14:50.892709 29876 tablet.cc:1722] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:50.896430 29893 ts_tablet_manager.cc:1888] T 56dddccc99524329bf2992a14c6235c7 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:50.897979 29893 log.cc:1196] T 56dddccc99524329bf2992a14c6235c7 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/wals/56dddccc99524329bf2992a14c6235c7
I20240502 14:14:50.898546 29893 ts_tablet_manager.cc:1909] T 56dddccc99524329bf2992a14c6235c7 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting consensus metadata
I20240502 14:14:50.898835 29783 ts_tablet_manager.cc:1888] T 64af9832dbc0464d8eea5f434172ee1f P e4cfb8fc5a474177af6848a6efa13a27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:50.901157 29783 log.cc:1196] T 64af9832dbc0464d8eea5f434172ee1f P e4cfb8fc5a474177af6848a6efa13a27: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/wals/64af9832dbc0464d8eea5f434172ee1f
I20240502 14:14:50.904232 29783 ts_tablet_manager.cc:1909] T 64af9832dbc0464d8eea5f434172ee1f P e4cfb8fc5a474177af6848a6efa13a27: Deleting consensus metadata
I20240502 14:14:50.907519 26272 catalog_manager.cc:4830] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): tablet 56dddccc99524329bf2992a14c6235c7 (table rebalance_test_table_8 [id=a0c39b03bebf4af19fe58980960cc466]) successfully deleted
I20240502 14:14:50.907954 29893 tablet_replica.cc:330] T 64af9832dbc0464d8eea5f434172ee1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: stopping tablet replica
I20240502 14:14:50.908548 29893 raft_consensus.cc:2229] T 64af9832dbc0464d8eea5f434172ee1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:50.909511 26272 catalog_manager.cc:4830] TS e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): tablet 64af9832dbc0464d8eea5f434172ee1f (table rebalance_test_table_8 [id=a0c39b03bebf4af19fe58980960cc466]) successfully deleted
I20240502 14:14:50.913048 29893 raft_consensus.cc:2258] T 64af9832dbc0464d8eea5f434172ee1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:50.914116 29893 ts_tablet_manager.cc:1875] T 64af9832dbc0464d8eea5f434172ee1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:50.920060 29782 ts_tablet_manager.cc:1888] T 56dddccc99524329bf2992a14c6235c7 P 89af7f423cf44717a08d6b142f0ed49d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:50.921947 29782 log.cc:1196] T 56dddccc99524329bf2992a14c6235c7 P 89af7f423cf44717a08d6b142f0ed49d: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/56dddccc99524329bf2992a14c6235c7
I20240502 14:14:50.922788 29782 ts_tablet_manager.cc:1909] T 56dddccc99524329bf2992a14c6235c7 P 89af7f423cf44717a08d6b142f0ed49d: Deleting consensus metadata
I20240502 14:14:50.924003 29782 tablet_replica.cc:330] T 64af9832dbc0464d8eea5f434172ee1f P 89af7f423cf44717a08d6b142f0ed49d: stopping tablet replica
I20240502 14:14:50.924130 26271 catalog_manager.cc:4830] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): tablet 56dddccc99524329bf2992a14c6235c7 (table rebalance_test_table_8 [id=a0c39b03bebf4af19fe58980960cc466]) successfully deleted
I20240502 14:14:50.924345 29782 raft_consensus.cc:2229] T 64af9832dbc0464d8eea5f434172ee1f P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:50.924762 29782 raft_consensus.cc:2258] T 64af9832dbc0464d8eea5f434172ee1f P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:50.926577 29893 ts_tablet_manager.cc:1888] T 64af9832dbc0464d8eea5f434172ee1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:50.926851 29893 log.cc:1196] T 64af9832dbc0464d8eea5f434172ee1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/wals/64af9832dbc0464d8eea5f434172ee1f
I20240502 14:14:50.927364 29893 ts_tablet_manager.cc:1909] T 64af9832dbc0464d8eea5f434172ee1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting consensus metadata
I20240502 14:14:50.927718 29782 ts_tablet_manager.cc:1875] T 64af9832dbc0464d8eea5f434172ee1f P 89af7f423cf44717a08d6b142f0ed49d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:50.928648 26272 catalog_manager.cc:4830] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): tablet 64af9832dbc0464d8eea5f434172ee1f (table rebalance_test_table_8 [id=a0c39b03bebf4af19fe58980960cc466]) successfully deleted
I20240502 14:14:50.930298 26277 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_extra_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:50.939996 29782 ts_tablet_manager.cc:1888] T 64af9832dbc0464d8eea5f434172ee1f P 89af7f423cf44717a08d6b142f0ed49d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:50.940491 29782 log.cc:1196] T 64af9832dbc0464d8eea5f434172ee1f P 89af7f423cf44717a08d6b142f0ed49d: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/64af9832dbc0464d8eea5f434172ee1f
I20240502 14:14:50.941013 29289 tablet_service.cc:1467] Processing CreateTablet for tablet cf808a060536445495a70de638438f87 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=052077f1ff8540afbb74e3609ab56e61]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:50.941483 29289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf808a060536445495a70de638438f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:50.942802 29423 tablet_service.cc:1467] Processing CreateTablet for tablet 1704fbfbeed04dcaac8bc663b7ee2a5b (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=052077f1ff8540afbb74e3609ab56e61]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:50.943265 29423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1704fbfbeed04dcaac8bc663b7ee2a5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:50.947592 29555 tablet_service.cc:1467] Processing CreateTablet for tablet 1704fbfbeed04dcaac8bc663b7ee2a5b (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=052077f1ff8540afbb74e3609ab56e61]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:50.948069 29555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1704fbfbeed04dcaac8bc663b7ee2a5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:50.949054 26537 tablet_service.cc:1467] Processing CreateTablet for tablet 39e6a71b3eef40848ea1ba4ef9604fa2 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=052077f1ff8540afbb74e3609ab56e61]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:50.949433 29155 tablet_service.cc:1467] Processing CreateTablet for tablet 39e6a71b3eef40848ea1ba4ef9604fa2 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=052077f1ff8540afbb74e3609ab56e61]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:50.949589 26537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39e6a71b3eef40848ea1ba4ef9604fa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:50.949882 29155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39e6a71b3eef40848ea1ba4ef9604fa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:50.952996 29287 tablet_service.cc:1467] Processing CreateTablet for tablet 1704fbfbeed04dcaac8bc663b7ee2a5b (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=052077f1ff8540afbb74e3609ab56e61]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:50.953572 29287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1704fbfbeed04dcaac8bc663b7ee2a5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:50.957660 29288 tablet_service.cc:1467] Processing CreateTablet for tablet 39e6a71b3eef40848ea1ba4ef9604fa2 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=052077f1ff8540afbb74e3609ab56e61]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:50.958160 29288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39e6a71b3eef40848ea1ba4ef9604fa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:50.942803 29421 tablet_service.cc:1467] Processing CreateTablet for tablet cf808a060536445495a70de638438f87 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=052077f1ff8540afbb74e3609ab56e61]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:50.962724 29421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf808a060536445495a70de638438f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:50.965286 29894 tablet_bootstrap.cc:492] T 39e6a71b3eef40848ea1ba4ef9604fa2 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:50.947041 26406 tablet_service.cc:1467] Processing CreateTablet for tablet cf808a060536445495a70de638438f87 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=052077f1ff8540afbb74e3609ab56e61]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:50.968353 29897 tablet_bootstrap.cc:492] T cf808a060536445495a70de638438f87 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:14:50.968593 26406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf808a060536445495a70de638438f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:50.970844 29897 tablet_bootstrap.cc:654] T cf808a060536445495a70de638438f87 P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:50.971594 29894 tablet_bootstrap.cc:654] T 39e6a71b3eef40848ea1ba4ef9604fa2 P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:50.973021 29898 tablet_bootstrap.cc:492] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:14:50.979198 29782 ts_tablet_manager.cc:1909] T 64af9832dbc0464d8eea5f434172ee1f P 89af7f423cf44717a08d6b142f0ed49d: Deleting consensus metadata
I20240502 14:14:50.979869 29897 tablet_bootstrap.cc:492] T cf808a060536445495a70de638438f87 P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:14:50.980187 29897 ts_tablet_manager.cc:1367] T cf808a060536445495a70de638438f87 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20240502 14:14:50.981186 29897 raft_consensus.cc:348] T cf808a060536445495a70de638438f87 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:50.981611 29897 raft_consensus.cc:374] T cf808a060536445495a70de638438f87 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:50.982096 29897 raft_consensus.cc:729] T cf808a060536445495a70de638438f87 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:14:50.982530 29897 consensus_queue.cc:260] T cf808a060536445495a70de638438f87 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:50.983309 29897 ts_tablet_manager.cc:1398] T cf808a060536445495a70de638438f87 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:14:50.983693 29897 tablet_bootstrap.cc:492] T 39e6a71b3eef40848ea1ba4ef9604fa2 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:14:50.983690 26271 catalog_manager.cc:4830] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): tablet 64af9832dbc0464d8eea5f434172ee1f (table rebalance_test_table_8 [id=a0c39b03bebf4af19fe58980960cc466]) successfully deleted
I20240502 14:14:50.985427 29894 tablet_bootstrap.cc:492] T 39e6a71b3eef40848ea1ba4ef9604fa2 P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:50.985821 29897 tablet_bootstrap.cc:654] T 39e6a71b3eef40848ea1ba4ef9604fa2 P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:50.985855 29894 ts_tablet_manager.cc:1367] T 39e6a71b3eef40848ea1ba4ef9604fa2 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.000s
I20240502 14:14:50.986547 29898 tablet_bootstrap.cc:654] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:50.986685 29902 tablet_bootstrap.cc:492] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:50.988667 29894 raft_consensus.cc:348] T 39e6a71b3eef40848ea1ba4ef9604fa2 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:50.990303 29894 raft_consensus.cc:374] T 39e6a71b3eef40848ea1ba4ef9604fa2 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:50.990433 29902 tablet_bootstrap.cc:654] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:50.992533 29894 raft_consensus.cc:729] T 39e6a71b3eef40848ea1ba4ef9604fa2 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:50.991300 29898 tablet_bootstrap.cc:492] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:14:50.992895 29898 ts_tablet_manager.cc:1367] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.000s
I20240502 14:14:50.993520 29894 consensus_queue.cc:260] T 39e6a71b3eef40848ea1ba4ef9604fa2 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:50.994131 26277 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:14:50.994352 26277 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:14:50.995369 29894 ts_tablet_manager.cc:1398] T 39e6a71b3eef40848ea1ba4ef9604fa2 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 14:14:50.994125 29898 raft_consensus.cc:348] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:50.995481 29898 raft_consensus.cc:374] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:50.995587 29898 raft_consensus.cc:729] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:14:50.995849 29898 consensus_queue.cc:260] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:50.996480 29898 ts_tablet_manager.cc:1398] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:50.999032 29897 tablet_bootstrap.cc:492] T 39e6a71b3eef40848ea1ba4ef9604fa2 P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:14:50.999508 29897 ts_tablet_manager.cc:1367] T 39e6a71b3eef40848ea1ba4ef9604fa2 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.004s
I20240502 14:14:51.000286 29897 raft_consensus.cc:348] T 39e6a71b3eef40848ea1ba4ef9604fa2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:51.000735 29897 raft_consensus.cc:374] T 39e6a71b3eef40848ea1ba4ef9604fa2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:51.000870 29897 raft_consensus.cc:729] T 39e6a71b3eef40848ea1ba4ef9604fa2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:14:51.001233 29897 consensus_queue.cc:260] T 39e6a71b3eef40848ea1ba4ef9604fa2 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:51.001386 26277 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet ca7bf077ff4945089c6cc9cdc1d1b5f9
I20240502 14:14:51.001837 29897 ts_tablet_manager.cc:1398] T 39e6a71b3eef40848ea1ba4ef9604fa2 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:51.002468 29555 tablet_service.cc:1514] Processing DeleteTablet for tablet ca7bf077ff4945089c6cc9cdc1d1b5f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:36398
I20240502 14:14:51.007072 29909 tablet_replica.cc:330] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P 4092902b3e674857a0147f1b4f2cdb86: stopping tablet replica
I20240502 14:14:51.007404 29909 raft_consensus.cc:2229] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:51.007683 29909 raft_consensus.cc:2258] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:51.009197 29909 ts_tablet_manager.cc:1875] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P 4092902b3e674857a0147f1b4f2cdb86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:51.014667 29897 tablet_bootstrap.cc:492] T 1704fbfbeed04dcaac8bc663b7ee2a5b P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:14:51.016656 29897 tablet_bootstrap.cc:654] T 1704fbfbeed04dcaac8bc663b7ee2a5b P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:51.017680 29909 ts_tablet_manager.cc:1888] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P 4092902b3e674857a0147f1b4f2cdb86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:51.017889 29909 log.cc:1196] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P 4092902b3e674857a0147f1b4f2cdb86: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/wal/wals/ca7bf077ff4945089c6cc9cdc1d1b5f9
I20240502 14:14:51.018416 29909 ts_tablet_manager.cc:1909] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P 4092902b3e674857a0147f1b4f2cdb86: Deleting consensus metadata
I20240502 14:14:51.024828 26272 catalog_manager.cc:4830] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097): tablet ca7bf077ff4945089c6cc9cdc1d1b5f9 (table rebalancer_dynamic_table_new_name [id=fddfe01d8d2b414896e3001ff997b4d2]) successfully deleted
I20240502 14:14:51.033074 29912 tablet_bootstrap.cc:492] T 39e6a71b3eef40848ea1ba4ef9604fa2 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:14:51.034181 26537 tablet_service.cc:1514] Processing DeleteTablet for tablet ca7bf077ff4945089c6cc9cdc1d1b5f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:51696
I20240502 14:14:51.034576 29781 tablet_replica.cc:330] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P ba187e38a2754d60a12de9ac4856430a: stopping tablet replica
I20240502 14:14:51.034819 29781 raft_consensus.cc:2229] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:51.035079 29781 raft_consensus.cc:2258] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:51.036314 29781 ts_tablet_manager.cc:1875] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P ba187e38a2754d60a12de9ac4856430a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:51.040091 29897 tablet_bootstrap.cc:492] T 1704fbfbeed04dcaac8bc663b7ee2a5b P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:14:51.040263 29897 ts_tablet_manager.cc:1367] T 1704fbfbeed04dcaac8bc663b7ee2a5b P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.003s
I20240502 14:14:51.041280 29897 raft_consensus.cc:348] T 1704fbfbeed04dcaac8bc663b7ee2a5b P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:51.041559 29897 raft_consensus.cc:374] T 1704fbfbeed04dcaac8bc663b7ee2a5b P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:51.041687 29897 raft_consensus.cc:729] T 1704fbfbeed04dcaac8bc663b7ee2a5b P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:14:51.041966 29897 consensus_queue.cc:260] T 1704fbfbeed04dcaac8bc663b7ee2a5b P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:51.043408 29781 ts_tablet_manager.cc:1888] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P ba187e38a2754d60a12de9ac4856430a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:51.043641 29781 log.cc:1196] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P ba187e38a2754d60a12de9ac4856430a: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/ca7bf077ff4945089c6cc9cdc1d1b5f9
I20240502 14:14:51.044214 29781 ts_tablet_manager.cc:1909] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P ba187e38a2754d60a12de9ac4856430a: Deleting consensus metadata
I20240502 14:14:51.045969 29897 ts_tablet_manager.cc:1398] T 1704fbfbeed04dcaac8bc663b7ee2a5b P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 14:14:51.046249 29288 tablet_service.cc:1514] Processing DeleteTablet for tablet ca7bf077ff4945089c6cc9cdc1d1b5f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:59160
I20240502 14:14:51.049103 29912 tablet_bootstrap.cc:654] T 39e6a71b3eef40848ea1ba4ef9604fa2 P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:51.053206 26271 catalog_manager.cc:4830] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): tablet ca7bf077ff4945089c6cc9cdc1d1b5f9 (table rebalancer_dynamic_table_new_name [id=fddfe01d8d2b414896e3001ff997b4d2]) successfully deleted
I20240502 14:14:51.058205 29913 tablet_replica.cc:330] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P c1100e16b3b9495ca61d4f95e89b6a89: stopping tablet replica
I20240502 14:14:51.058554 29913 raft_consensus.cc:2229] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:51.059355 29913 raft_consensus.cc:2258] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:51.061383 29913 ts_tablet_manager.cc:1875] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:51.069327 29910 tablet_bootstrap.cc:492] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:14:51.070626 29912 tablet_bootstrap.cc:492] T 39e6a71b3eef40848ea1ba4ef9604fa2 P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:14:51.071353 29910 tablet_bootstrap.cc:654] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:51.070904 29912 ts_tablet_manager.cc:1367] T 39e6a71b3eef40848ea1ba4ef9604fa2 P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.038s user 0.005s sys 0.000s
I20240502 14:14:51.073740 29902 tablet_bootstrap.cc:492] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:51.073988 29902 ts_tablet_manager.cc:1367] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.090s user 0.011s sys 0.000s
I20240502 14:14:51.074510 29912 raft_consensus.cc:348] T 39e6a71b3eef40848ea1ba4ef9604fa2 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:51.074977 29902 raft_consensus.cc:348] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:51.076009 29912 raft_consensus.cc:374] T 39e6a71b3eef40848ea1ba4ef9604fa2 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:51.076072 29902 raft_consensus.cc:374] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:51.076192 29902 raft_consensus.cc:729] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:51.076231 29912 raft_consensus.cc:729] T 39e6a71b3eef40848ea1ba4ef9604fa2 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:14:51.076545 29902 consensus_queue.cc:260] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:51.076840 29912 consensus_queue.cc:260] T 39e6a71b3eef40848ea1ba4ef9604fa2 P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:51.077286 29902 ts_tablet_manager.cc:1398] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:51.087220 29910 tablet_bootstrap.cc:492] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:14:51.087635 29910 ts_tablet_manager.cc:1367] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.000s
I20240502 14:14:51.089298 29913 ts_tablet_manager.cc:1888] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P c1100e16b3b9495ca61d4f95e89b6a89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:51.090480 29912 ts_tablet_manager.cc:1398] T 39e6a71b3eef40848ea1ba4ef9604fa2 P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.019s user 0.003s sys 0.000s
I20240502 14:14:51.090797 29913 log.cc:1196] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal/wals/ca7bf077ff4945089c6cc9cdc1d1b5f9
I20240502 14:14:51.091356 29913 ts_tablet_manager.cc:1909] T ca7bf077ff4945089c6cc9cdc1d1b5f9 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting consensus metadata
I20240502 14:14:51.093977 26271 catalog_manager.cc:4830] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): tablet ca7bf077ff4945089c6cc9cdc1d1b5f9 (table rebalancer_dynamic_table_new_name [id=fddfe01d8d2b414896e3001ff997b4d2]) successfully deleted
I20240502 14:14:51.101002 29918 raft_consensus.cc:482] T 39e6a71b3eef40848ea1ba4ef9604fa2 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:51.101320 29918 raft_consensus.cc:504] T 39e6a71b3eef40848ea1ba4ef9604fa2 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:51.105470 26283 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:14:51.105834 29910 raft_consensus.cc:348] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:51.108011 29910 raft_consensus.cc:374] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:51.108611 29910 raft_consensus.cc:729] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:14:51.108563 29918 leader_election.cc:288] T 39e6a71b3eef40848ea1ba4ef9604fa2 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:14:51.109409 29910 consensus_queue.cc:260] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:51.114756 29910 ts_tablet_manager.cc:1398] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.027s user 0.003s sys 0.000s
I20240502 14:14:51.115285 29910 tablet_bootstrap.cc:492] T cf808a060536445495a70de638438f87 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:14:51.125162 29910 tablet_bootstrap.cc:654] T cf808a060536445495a70de638438f87 P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:51.131996 29155 tablet_service.cc:1467] Processing CreateTablet for tablet edc8fa36097c49c88c1b5b981a32e2a2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=02258aeb7420405ab4cce9f74610170f]), partition=
I20240502 14:14:51.132081 26797 tablet_service.cc:1467] Processing CreateTablet for tablet edc8fa36097c49c88c1b5b981a32e2a2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=02258aeb7420405ab4cce9f74610170f]), partition=
I20240502 14:14:51.132531 29155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edc8fa36097c49c88c1b5b981a32e2a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:51.132567 26797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edc8fa36097c49c88c1b5b981a32e2a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:51.136838 26406 tablet_service.cc:1467] Processing CreateTablet for tablet edc8fa36097c49c88c1b5b981a32e2a2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=02258aeb7420405ab4cce9f74610170f]), partition=
I20240502 14:14:51.137290 26406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edc8fa36097c49c88c1b5b981a32e2a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:51.147516 29912 tablet_bootstrap.cc:492] T edc8fa36097c49c88c1b5b981a32e2a2 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:14:51.149008 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e6a71b3eef40848ea1ba4ef9604fa2" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:51.149293 26558 raft_consensus.cc:2454] T 39e6a71b3eef40848ea1ba4ef9604fa2 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 0.
I20240502 14:14:51.149921 29902 tablet_bootstrap.cc:492] T edc8fa36097c49c88c1b5b981a32e2a2 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:51.150224 29111 leader_election.cc:302] T 39e6a71b3eef40848ea1ba4ef9604fa2 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:51.150667 29918 raft_consensus.cc:2781] T 39e6a71b3eef40848ea1ba4ef9604fa2 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:51.150771 29918 raft_consensus.cc:482] T 39e6a71b3eef40848ea1ba4ef9604fa2 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:51.150846 29918 raft_consensus.cc:3037] T 39e6a71b3eef40848ea1ba4ef9604fa2 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:51.152244 29902 tablet_bootstrap.cc:654] T edc8fa36097c49c88c1b5b981a32e2a2 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:51.153985 29918 raft_consensus.cc:504] T 39e6a71b3eef40848ea1ba4ef9604fa2 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:51.157567 29912 tablet_bootstrap.cc:654] T edc8fa36097c49c88c1b5b981a32e2a2 P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:51.157749 29918 leader_election.cc:288] T 39e6a71b3eef40848ea1ba4ef9604fa2 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 election: Requested vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:14:51.158558 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e6a71b3eef40848ea1ba4ef9604fa2" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:14:51.158816 26558 raft_consensus.cc:3037] T 39e6a71b3eef40848ea1ba4ef9604fa2 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:51.159425 29926 tablet_bootstrap.cc:492] T edc8fa36097c49c88c1b5b981a32e2a2 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:51.165764 29902 tablet_bootstrap.cc:492] T edc8fa36097c49c88c1b5b981a32e2a2 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:51.165989 29902 ts_tablet_manager.cc:1367] T edc8fa36097c49c88c1b5b981a32e2a2 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.006s
I20240502 14:14:51.166286 29926 tablet_bootstrap.cc:654] T edc8fa36097c49c88c1b5b981a32e2a2 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:51.166604 29912 tablet_bootstrap.cc:492] T edc8fa36097c49c88c1b5b981a32e2a2 P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:14:51.166864 29912 ts_tablet_manager.cc:1367] T edc8fa36097c49c88c1b5b981a32e2a2 P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.000s
I20240502 14:14:51.167006 29902 raft_consensus.cc:348] T edc8fa36097c49c88c1b5b981a32e2a2 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:51.167269 29902 raft_consensus.cc:374] T edc8fa36097c49c88c1b5b981a32e2a2 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:51.167366 29902 raft_consensus.cc:729] T edc8fa36097c49c88c1b5b981a32e2a2 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:51.167683 29902 consensus_queue.cc:260] T edc8fa36097c49c88c1b5b981a32e2a2 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:51.168280 29902 ts_tablet_manager.cc:1398] T edc8fa36097c49c88c1b5b981a32e2a2 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:51.168820 29912 raft_consensus.cc:348] T edc8fa36097c49c88c1b5b981a32e2a2 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:51.169090 29912 raft_consensus.cc:374] T edc8fa36097c49c88c1b5b981a32e2a2 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:51.169185 29912 raft_consensus.cc:729] T edc8fa36097c49c88c1b5b981a32e2a2 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:14:51.169466 29912 consensus_queue.cc:260] T edc8fa36097c49c88c1b5b981a32e2a2 P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:51.170033 29912 ts_tablet_manager.cc:1398] T edc8fa36097c49c88c1b5b981a32e2a2 P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:51.171500 26558 raft_consensus.cc:2454] T 39e6a71b3eef40848ea1ba4ef9604fa2 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 1.
I20240502 14:14:51.172188 29111 leader_election.cc:302] T 39e6a71b3eef40848ea1ba4ef9604fa2 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:51.172717 29918 raft_consensus.cc:2781] T 39e6a71b3eef40848ea1ba4ef9604fa2 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:51.173234 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e6a71b3eef40848ea1ba4ef9604fa2" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:14:51.173532 29309 raft_consensus.cc:2454] T 39e6a71b3eef40848ea1ba4ef9604fa2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 0.
I20240502 14:14:51.173662 29918 raft_consensus.cc:686] T 39e6a71b3eef40848ea1ba4ef9604fa2 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 LEADER]: Becoming Leader. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Running, Role: LEADER
I20240502 14:14:51.173995 29308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e6a71b3eef40848ea1ba4ef9604fa2" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
I20240502 14:14:51.174204 29308 raft_consensus.cc:3037] T 39e6a71b3eef40848ea1ba4ef9604fa2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:51.174156 29918 consensus_queue.cc:237] T 39e6a71b3eef40848ea1ba4ef9604fa2 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:51.176776 29308 raft_consensus.cc:2454] T 39e6a71b3eef40848ea1ba4ef9604fa2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 1.
I20240502 14:14:51.177803 29910 tablet_bootstrap.cc:492] T cf808a060536445495a70de638438f87 P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:14:51.178021 29910 ts_tablet_manager.cc:1367] T cf808a060536445495a70de638438f87 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.063s user 0.004s sys 0.000s
I20240502 14:14:51.178987 29910 raft_consensus.cc:348] T cf808a060536445495a70de638438f87 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:51.179267 29910 raft_consensus.cc:374] T cf808a060536445495a70de638438f87 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:51.179363 29910 raft_consensus.cc:729] T cf808a060536445495a70de638438f87 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:14:51.179639 29910 consensus_queue.cc:260] T cf808a060536445495a70de638438f87 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:51.180233 29910 ts_tablet_manager.cc:1398] T cf808a060536445495a70de638438f87 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 14:14:51.181277 26277 catalog_manager.cc:5484] T 39e6a71b3eef40848ea1ba4ef9604fa2 P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197). New cstate: current_term: 1 leader_uuid: "1cb67158133e4648a6eceba32d4dbba3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:14:51.184888 29926 tablet_bootstrap.cc:492] T edc8fa36097c49c88c1b5b981a32e2a2 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:51.185400 29926 ts_tablet_manager.cc:1367] T edc8fa36097c49c88c1b5b981a32e2a2 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.002s
I20240502 14:14:51.186358 29926 raft_consensus.cc:348] T edc8fa36097c49c88c1b5b981a32e2a2 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:51.186703 29926 raft_consensus.cc:374] T edc8fa36097c49c88c1b5b981a32e2a2 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:51.186872 29926 raft_consensus.cc:729] T edc8fa36097c49c88c1b5b981a32e2a2 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:51.187220 29926 consensus_queue.cc:260] T edc8fa36097c49c88c1b5b981a32e2a2 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:51.188313 29926 ts_tablet_manager.cc:1398] T edc8fa36097c49c88c1b5b981a32e2a2 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:14:51.192932 29704 raft_consensus.cc:482] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:51.193118 29704 raft_consensus.cc:504] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:51.193874 29704 leader_election.cc:288] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:14:51.194622 29442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1704fbfbeed04dcaac8bc663b7ee2a5b" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:14:51.194845 29442 raft_consensus.cc:2454] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:14:51.195667 29511 leader_election.cc:302] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, 68a3c2b17ed94e06aba66ed4178132ab; no voters:
I20240502 14:14:51.195832 29308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1704fbfbeed04dcaac8bc663b7ee2a5b" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:14:51.196090 29704 raft_consensus.cc:2781] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:51.196102 29308 raft_consensus.cc:2454] T 1704fbfbeed04dcaac8bc663b7ee2a5b P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:14:51.196202 29704 raft_consensus.cc:482] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:51.196281 29704 raft_consensus.cc:3037] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:51.201190 29704 raft_consensus.cc:504] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:51.202723 29442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1704fbfbeed04dcaac8bc663b7ee2a5b" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
I20240502 14:14:51.202701 29308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1704fbfbeed04dcaac8bc663b7ee2a5b" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
I20240502 14:14:51.202939 29442 raft_consensus.cc:3037] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:51.203009 29308 raft_consensus.cc:3037] T 1704fbfbeed04dcaac8bc663b7ee2a5b P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:51.208612 29872 raft_consensus.cc:482] T cf808a060536445495a70de638438f87 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:51.208811 29872 raft_consensus.cc:504] T cf808a060536445495a70de638438f87 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:51.209775 29872 leader_election.cc:288] T cf808a060536445495a70de638438f87 P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:14:51.210203 29308 raft_consensus.cc:2454] T 1704fbfbeed04dcaac8bc663b7ee2a5b P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:14:51.210716 29442 raft_consensus.cc:2454] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:14:51.210808 29510 leader_election.cc:302] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:14:51.211019 29704 leader_election.cc:288] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Requested vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:14:51.211225 29704 raft_consensus.cc:2781] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:51.211329 29704 raft_consensus.cc:686] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Becoming Leader. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Running, Role: LEADER
I20240502 14:14:51.211608 29704 consensus_queue.cc:237] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:51.220571 29442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf808a060536445495a70de638438f87" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:14:51.234534 29442 raft_consensus.cc:2454] T cf808a060536445495a70de638438f87 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 0.
I20240502 14:14:51.235519 29245 leader_election.cc:302] T cf808a060536445495a70de638438f87 P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:14:51.236135 29872 raft_consensus.cc:2781] T cf808a060536445495a70de638438f87 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:51.236315 29872 raft_consensus.cc:482] T cf808a060536445495a70de638438f87 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:51.236447 29872 raft_consensus.cc:3037] T cf808a060536445495a70de638438f87 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:51.241407 29872 raft_consensus.cc:504] T cf808a060536445495a70de638438f87 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:51.246408 29872 leader_election.cc:288] T cf808a060536445495a70de638438f87 P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 election: Requested vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:14:51.247776 29442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf808a060536445495a70de638438f87" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
I20240502 14:14:51.250886 29442 raft_consensus.cc:3037] T cf808a060536445495a70de638438f87 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:51.266860 29442 raft_consensus.cc:2454] T cf808a060536445495a70de638438f87 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 1.
I20240502 14:14:51.267808 29245 leader_election.cc:302] T cf808a060536445495a70de638438f87 P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:14:51.268350 29872 raft_consensus.cc:2781] T cf808a060536445495a70de638438f87 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:51.269117 29872 raft_consensus.cc:686] T cf808a060536445495a70de638438f87 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 LEADER]: Becoming Leader. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Running, Role: LEADER
I20240502 14:14:51.269555 29872 consensus_queue.cc:237] T cf808a060536445495a70de638438f87 P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:51.290915 26278 catalog_manager.cc:5484] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200). New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:14:51.297504 29003 raft_consensus.cc:482] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:51.297683 29003 raft_consensus.cc:504] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:51.298565 29003 leader_election.cc:288] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:14:51.327553 26278 catalog_manager.cc:5484] T cf808a060536445495a70de638438f87 P c1100e16b3b9495ca61d4f95e89b6a89 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198). New cstate: current_term: 1 leader_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:14:51.329288 26277 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240502 14:14:51.329571 26277 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240502 14:14:51.333760 26277 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet c3ca60f2b4fa4852a9ec585c640fd21e
I20240502 14:14:51.335753 26669 tablet_service.cc:1514] Processing DeleteTablet for tablet c3ca60f2b4fa4852a9ec585c640fd21e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:51 UTC) from {username='slave'} at 127.0.0.1:34150
I20240502 14:14:51.336095 29893 tablet_replica.cc:330] T c3ca60f2b4fa4852a9ec585c640fd21e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: stopping tablet replica
I20240502 14:14:51.336337 29893 raft_consensus.cc:2229] T c3ca60f2b4fa4852a9ec585c640fd21e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:51.336773 29893 raft_consensus.cc:2258] T c3ca60f2b4fa4852a9ec585c640fd21e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:51.337023 26537 tablet_service.cc:1514] Processing DeleteTablet for tablet c3ca60f2b4fa4852a9ec585c640fd21e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:51 UTC) from {username='slave'} at 127.0.0.1:51696
I20240502 14:14:51.337301 26277 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet 6c5232e2b8ec4aba8bf2fe03aa132a6e
I20240502 14:14:51.337343 29781 tablet_replica.cc:330] T c3ca60f2b4fa4852a9ec585c640fd21e P ba187e38a2754d60a12de9ac4856430a: stopping tablet replica
I20240502 14:14:51.337579 29781 raft_consensus.cc:2229] T c3ca60f2b4fa4852a9ec585c640fd21e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:51.337610 29893 ts_tablet_manager.cc:1875] T c3ca60f2b4fa4852a9ec585c640fd21e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:51.337838 29781 raft_consensus.cc:2258] T c3ca60f2b4fa4852a9ec585c640fd21e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:51.338630 29781 ts_tablet_manager.cc:1875] T c3ca60f2b4fa4852a9ec585c640fd21e P ba187e38a2754d60a12de9ac4856430a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:51.339826 26277 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet 8d7c842c6c2e4929b07173ab4b7dc224
I20240502 14:14:51.340605 26669 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c5232e2b8ec4aba8bf2fe03aa132a6e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:51 UTC) from {username='slave'} at 127.0.0.1:34150
I20240502 14:14:51.340912 26668 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d7c842c6c2e4929b07173ab4b7dc224 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:51 UTC) from {username='slave'} at 127.0.0.1:34150
I20240502 14:14:51.341154 29226 raft_consensus.cc:482] T edc8fa36097c49c88c1b5b981a32e2a2 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:51.341321 29226 raft_consensus.cc:504] T edc8fa36097c49c88c1b5b981a32e2a2 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:51.342304 29226 leader_election.cc:288] T edc8fa36097c49c88c1b5b981a32e2a2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:14:51.343642 26797 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c5232e2b8ec4aba8bf2fe03aa132a6e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:51 UTC) from {username='slave'} at 127.0.0.1:56010
I20240502 14:14:51.344128 29783 tablet_replica.cc:330] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P e4cfb8fc5a474177af6848a6efa13a27: stopping tablet replica
I20240502 14:14:51.344549 29783 raft_consensus.cc:2229] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:51.345234 29783 raft_consensus.cc:2258] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:51.346101 29893 ts_tablet_manager.cc:1888] T c3ca60f2b4fa4852a9ec585c640fd21e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:51.346277 29893 log.cc:1196] T c3ca60f2b4fa4852a9ec585c640fd21e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/wals/c3ca60f2b4fa4852a9ec585c640fd21e
I20240502 14:14:51.346323 29783 ts_tablet_manager.cc:1875] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P e4cfb8fc5a474177af6848a6efa13a27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:51.346832 29893 ts_tablet_manager.cc:1909] T c3ca60f2b4fa4852a9ec585c640fd21e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting consensus metadata
I20240502 14:14:51.349049 26537 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d7c842c6c2e4929b07173ab4b7dc224 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:51 UTC) from {username='slave'} at 127.0.0.1:51696
I20240502 14:14:51.349395 26272 catalog_manager.cc:4830] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): tablet c3ca60f2b4fa4852a9ec585c640fd21e (table rebalance_test_table_10 [id=37d9dbdf887c42b984b9623e7fec549d]) successfully deleted
I20240502 14:14:51.349915 29893 tablet_replica.cc:330] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: stopping tablet replica
I20240502 14:14:51.349977 29781 ts_tablet_manager.cc:1888] T c3ca60f2b4fa4852a9ec585c640fd21e P ba187e38a2754d60a12de9ac4856430a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:51.350312 29781 log.cc:1196] T c3ca60f2b4fa4852a9ec585c640fd21e P ba187e38a2754d60a12de9ac4856430a: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/c3ca60f2b4fa4852a9ec585c640fd21e
I20240502 14:14:51.350298 29893 raft_consensus.cc:2229] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:51.350544 29893 raft_consensus.cc:2258] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:51.350883 29781 ts_tablet_manager.cc:1909] T c3ca60f2b4fa4852a9ec585c640fd21e P ba187e38a2754d60a12de9ac4856430a: Deleting consensus metadata
I20240502 14:14:51.351403 29893 ts_tablet_manager.cc:1875] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:51.352968 26271 catalog_manager.cc:4830] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): tablet c3ca60f2b4fa4852a9ec585c640fd21e (table rebalance_test_table_10 [id=37d9dbdf887c42b984b9623e7fec549d]) successfully deleted
I20240502 14:14:51.353376 29781 tablet_replica.cc:330] T 8d7c842c6c2e4929b07173ab4b7dc224 P ba187e38a2754d60a12de9ac4856430a: stopping tablet replica
I20240502 14:14:51.353895 29781 raft_consensus.cc:2229] T 8d7c842c6c2e4929b07173ab4b7dc224 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:51.354310 29781 raft_consensus.cc:2258] T 8d7c842c6c2e4929b07173ab4b7dc224 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:51.355396 29781 ts_tablet_manager.cc:1875] T 8d7c842c6c2e4929b07173ab4b7dc224 P ba187e38a2754d60a12de9ac4856430a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:51.358690 29893 ts_tablet_manager.cc:1888] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:51.358922 29893 log.cc:1196] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/wals/6c5232e2b8ec4aba8bf2fe03aa132a6e
I20240502 14:14:51.359114 26406 tablet_service.cc:1514] Processing DeleteTablet for tablet c3ca60f2b4fa4852a9ec585c640fd21e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:51 UTC) from {username='slave'} at 127.0.0.1:37422
I20240502 14:14:51.359409 29782 tablet_replica.cc:330] T c3ca60f2b4fa4852a9ec585c640fd21e P 89af7f423cf44717a08d6b142f0ed49d: stopping tablet replica
I20240502 14:14:51.359501 29893 ts_tablet_manager.cc:1909] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting consensus metadata
I20240502 14:14:51.359660 29782 raft_consensus.cc:2229] T c3ca60f2b4fa4852a9ec585c640fd21e P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:51.359936 29782 raft_consensus.cc:2258] T c3ca60f2b4fa4852a9ec585c640fd21e P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:51.360677 29893 tablet_replica.cc:330] T 8d7c842c6c2e4929b07173ab4b7dc224 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: stopping tablet replica
I20240502 14:14:51.360803 29782 ts_tablet_manager.cc:1875] T c3ca60f2b4fa4852a9ec585c640fd21e P 89af7f423cf44717a08d6b142f0ed49d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:51.360960 26272 catalog_manager.cc:4830] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): tablet 6c5232e2b8ec4aba8bf2fe03aa132a6e (table rebalance_test_table_10 [id=37d9dbdf887c42b984b9623e7fec549d]) successfully deleted
I20240502 14:14:51.361037 29893 raft_consensus.cc:2229] T 8d7c842c6c2e4929b07173ab4b7dc224 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:51.361399 29893 raft_consensus.cc:2258] T 8d7c842c6c2e4929b07173ab4b7dc224 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:51.362308 29893 ts_tablet_manager.cc:1875] T 8d7c842c6c2e4929b07173ab4b7dc224 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:51.364547 29781 ts_tablet_manager.cc:1888] T 8d7c842c6c2e4929b07173ab4b7dc224 P ba187e38a2754d60a12de9ac4856430a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:51.364833 29781 log.cc:1196] T 8d7c842c6c2e4929b07173ab4b7dc224 P ba187e38a2754d60a12de9ac4856430a: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/8d7c842c6c2e4929b07173ab4b7dc224
I20240502 14:14:51.364948 29783 ts_tablet_manager.cc:1888] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P e4cfb8fc5a474177af6848a6efa13a27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:51.365155 29783 log.cc:1196] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P e4cfb8fc5a474177af6848a6efa13a27: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/wals/6c5232e2b8ec4aba8bf2fe03aa132a6e
I20240502 14:14:51.366240 29781 ts_tablet_manager.cc:1909] T 8d7c842c6c2e4929b07173ab4b7dc224 P ba187e38a2754d60a12de9ac4856430a: Deleting consensus metadata
I20240502 14:14:51.367587 29783 ts_tablet_manager.cc:1909] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P e4cfb8fc5a474177af6848a6efa13a27: Deleting consensus metadata
I20240502 14:14:51.367800 29782 ts_tablet_manager.cc:1888] T c3ca60f2b4fa4852a9ec585c640fd21e P 89af7f423cf44717a08d6b142f0ed49d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:51.367995 29782 log.cc:1196] T c3ca60f2b4fa4852a9ec585c640fd21e P 89af7f423cf44717a08d6b142f0ed49d: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/c3ca60f2b4fa4852a9ec585c640fd21e
I20240502 14:14:51.368484 29782 ts_tablet_manager.cc:1909] T c3ca60f2b4fa4852a9ec585c640fd21e P 89af7f423cf44717a08d6b142f0ed49d: Deleting consensus metadata
I20240502 14:14:51.368888 26406 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c5232e2b8ec4aba8bf2fe03aa132a6e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:51 UTC) from {username='slave'} at 127.0.0.1:37422
I20240502 14:14:51.368945 26271 catalog_manager.cc:4830] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): tablet 8d7c842c6c2e4929b07173ab4b7dc224 (table rebalance_test_table_10 [id=37d9dbdf887c42b984b9623e7fec549d]) successfully deleted
I20240502 14:14:51.369045 29893 ts_tablet_manager.cc:1888] T 8d7c842c6c2e4929b07173ab4b7dc224 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:51.369166 29893 log.cc:1196] T 8d7c842c6c2e4929b07173ab4b7dc224 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/wals/8d7c842c6c2e4929b07173ab4b7dc224
I20240502 14:14:51.369155 26407 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d7c842c6c2e4929b07173ab4b7dc224 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:51 UTC) from {username='slave'} at 127.0.0.1:37422
I20240502 14:14:51.369638 29893 ts_tablet_manager.cc:1909] T 8d7c842c6c2e4929b07173ab4b7dc224 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting consensus metadata
I20240502 14:14:51.369916 29308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf808a060536445495a70de638438f87" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:14:51.370465 29782 tablet_replica.cc:330] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P 89af7f423cf44717a08d6b142f0ed49d: stopping tablet replica
I20240502 14:14:51.370724 29782 raft_consensus.cc:2229] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:51.370807 26272 catalog_manager.cc:4830] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): tablet 8d7c842c6c2e4929b07173ab4b7dc224 (table rebalance_test_table_10 [id=37d9dbdf887c42b984b9623e7fec549d]) successfully deleted
I20240502 14:14:51.371052 29782 raft_consensus.cc:2258] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:51.372670 26272 catalog_manager.cc:4830] TS e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): tablet 6c5232e2b8ec4aba8bf2fe03aa132a6e (table rebalance_test_table_10 [id=37d9dbdf887c42b984b9623e7fec549d]) successfully deleted
I20240502 14:14:51.372586 29782 ts_tablet_manager.cc:1875] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P 89af7f423cf44717a08d6b142f0ed49d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:51.373018 26271 catalog_manager.cc:4830] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): tablet c3ca60f2b4fa4852a9ec585c640fd21e (table rebalance_test_table_10 [id=37d9dbdf887c42b984b9623e7fec549d]) successfully deleted
I20240502 14:14:51.375746 26423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc8fa36097c49c88c1b5b981a32e2a2" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:14:51.376009 26423 raft_consensus.cc:2454] T edc8fa36097c49c88c1b5b981a32e2a2 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 0.
I20240502 14:14:51.377342 29442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf808a060536445495a70de638438f87" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:14:51.377736 29442 raft_consensus.cc:2379] T cf808a060536445495a70de638438f87 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 89af7f423cf44717a08d6b142f0ed49d in current term 1: Already voted for candidate c1100e16b3b9495ca61d4f95e89b6a89 in this term.
I20240502 14:14:51.378693 26427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf808a060536445495a70de638438f87" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:14:51.378907 26427 raft_consensus.cc:3037] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:51.379726 26423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf808a060536445495a70de638438f87" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:14:51.380023 26754 leader_election.cc:302] T edc8fa36097c49c88c1b5b981a32e2a2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:51.380919 29226 raft_consensus.cc:2781] T edc8fa36097c49c88c1b5b981a32e2a2 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:51.381052 29226 raft_consensus.cc:482] T edc8fa36097c49c88c1b5b981a32e2a2 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:51.381165 29226 raft_consensus.cc:3037] T edc8fa36097c49c88c1b5b981a32e2a2 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:51.382913 26363 leader_election.cc:302] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
W20240502 14:14:51.383524 26363 consensus_peers.cc:485] T 8d7c842c6c2e4929b07173ab4b7dc224 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): Couldn't send request to peer 69ab78b2a1b74d2bbf2cdeafa0a4af23. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8d7c842c6c2e4929b07173ab4b7dc224. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:51.383901 29226 raft_consensus.cc:504] T edc8fa36097c49c88c1b5b981a32e2a2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:51.384665 29226 leader_election.cc:288] T edc8fa36097c49c88c1b5b981a32e2a2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Requested vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:14:51.385344 26423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc8fa36097c49c88c1b5b981a32e2a2" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:14:51.385584 26423 raft_consensus.cc:3037] T edc8fa36097c49c88c1b5b981a32e2a2 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:51.386763 29918 raft_consensus.cc:482] T edc8fa36097c49c88c1b5b981a32e2a2 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:51.386979 29918 raft_consensus.cc:504] T edc8fa36097c49c88c1b5b981a32e2a2 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:51.388208 29918 leader_election.cc:288] T edc8fa36097c49c88c1b5b981a32e2a2 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:14:51.388243 29782 ts_tablet_manager.cc:1888] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P 89af7f423cf44717a08d6b142f0ed49d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:51.388383 29782 log.cc:1196] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P 89af7f423cf44717a08d6b142f0ed49d: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/6c5232e2b8ec4aba8bf2fe03aa132a6e
I20240502 14:14:51.389041 26427 raft_consensus.cc:2454] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 1.
I20240502 14:14:51.389350 29782 ts_tablet_manager.cc:1909] T 6c5232e2b8ec4aba8bf2fe03aa132a6e P 89af7f423cf44717a08d6b142f0ed49d: Deleting consensus metadata
I20240502 14:14:51.390067 29940 raft_consensus.cc:2726] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:14:51.390897 26423 raft_consensus.cc:2454] T edc8fa36097c49c88c1b5b981a32e2a2 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
I20240502 14:14:51.392040 29782 tablet_replica.cc:330] T 8d7c842c6c2e4929b07173ab4b7dc224 P 89af7f423cf44717a08d6b142f0ed49d: stopping tablet replica
I20240502 14:14:51.392469 29782 raft_consensus.cc:2229] T 8d7c842c6c2e4929b07173ab4b7dc224 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:51.393074 29782 raft_consensus.cc:2258] T 8d7c842c6c2e4929b07173ab4b7dc224 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:51.393194 26754 leader_election.cc:302] T edc8fa36097c49c88c1b5b981a32e2a2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:51.393227 26271 catalog_manager.cc:4830] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): tablet 6c5232e2b8ec4aba8bf2fe03aa132a6e (table rebalance_test_table_10 [id=37d9dbdf887c42b984b9623e7fec549d]) successfully deleted
I20240502 14:14:51.393551 29226 raft_consensus.cc:2781] T edc8fa36097c49c88c1b5b981a32e2a2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:51.393790 29226 raft_consensus.cc:686] T edc8fa36097c49c88c1b5b981a32e2a2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Becoming Leader. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Running, Role: LEADER
I20240502 14:14:51.394225 29226 consensus_queue.cc:237] T edc8fa36097c49c88c1b5b981a32e2a2 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:51.397282 29782 ts_tablet_manager.cc:1875] T 8d7c842c6c2e4929b07173ab4b7dc224 P 89af7f423cf44717a08d6b142f0ed49d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:51.399216 26423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc8fa36097c49c88c1b5b981a32e2a2" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:14:51.399590 26423 raft_consensus.cc:2379] T edc8fa36097c49c88c1b5b981a32e2a2 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb67158133e4648a6eceba32d4dbba3 in current term 1: Already voted for candidate e4cfb8fc5a474177af6848a6efa13a27 in this term.
I20240502 14:14:51.402429 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc8fa36097c49c88c1b5b981a32e2a2" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:14:51.402892 29176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc8fa36097c49c88c1b5b981a32e2a2" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:14:51.403141 29112 leader_election.cc:302] T edc8fa36097c49c88c1b5b981a32e2a2 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3; no voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27
I20240502 14:14:51.403421 29174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc8fa36097c49c88c1b5b981a32e2a2" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:14:51.403190 29176 raft_consensus.cc:2454] T edc8fa36097c49c88c1b5b981a32e2a2 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 0.
I20240502 14:14:51.404033 29918 raft_consensus.cc:3037] T edc8fa36097c49c88c1b5b981a32e2a2 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:51.404825 29782 ts_tablet_manager.cc:1888] T 8d7c842c6c2e4929b07173ab4b7dc224 P 89af7f423cf44717a08d6b142f0ed49d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:51.404974 29782 log.cc:1196] T 8d7c842c6c2e4929b07173ab4b7dc224 P 89af7f423cf44717a08d6b142f0ed49d: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/8d7c842c6c2e4929b07173ab4b7dc224
I20240502 14:14:51.405328 29782 ts_tablet_manager.cc:1909] T 8d7c842c6c2e4929b07173ab4b7dc224 P 89af7f423cf44717a08d6b142f0ed49d: Deleting consensus metadata
I20240502 14:14:51.407121 26271 catalog_manager.cc:4830] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): tablet 8d7c842c6c2e4929b07173ab4b7dc224 (table rebalance_test_table_10 [id=37d9dbdf887c42b984b9623e7fec549d]) successfully deleted
I20240502 14:14:51.407310 29918 raft_consensus.cc:2726] T edc8fa36097c49c88c1b5b981a32e2a2 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:14:51.408901 26278 catalog_manager.cc:5484] T edc8fa36097c49c88c1b5b981a32e2a2 P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196). New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:14:51.575911 26283 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:14:51.587811 26423 raft_consensus.cc:1261] T edc8fa36097c49c88c1b5b981a32e2a2 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:51.587975 29176 raft_consensus.cc:1261] T edc8fa36097c49c88c1b5b981a32e2a2 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:51.588788 29943 consensus_queue.cc:1035] T edc8fa36097c49c88c1b5b981a32e2a2 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:51.589458 29943 consensus_queue.cc:1035] T edc8fa36097c49c88c1b5b981a32e2a2 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:51.589843 29704 consensus_queue.cc:1035] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:51.600970 29155 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:14:51.661648 29555 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:14:51.661908 29423 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:51.674810 29918 consensus_queue.cc:1035] T 39e6a71b3eef40848ea1ba4ef9604fa2 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:51.684131 26668 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:14:51.686208 29872 consensus_queue.cc:1035] T cf808a060536445495a70de638438f87 P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:51.713547 29872 consensus_queue.cc:1035] T cf808a060536445495a70de638438f87 P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:51.721552 29694 consensus_queue.cc:1035] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:51.732641 26283 catalog_manager.cc:5576] TS 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715) does not have the latest schema for tablet edc8fa36097c49c88c1b5b981a32e2a2 (table rebalancer_dynamic_table [id=02258aeb7420405ab4cce9f74610170f]). Expected version 1 got 0
I20240502 14:14:51.804726 26537 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:14:51.806648 29995 tablet.cc:1722] T edc8fa36097c49c88c1b5b981a32e2a2 P e4cfb8fc5a474177af6848a6efa13a27: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:51.811812 29995 tablet.cc:1620] T edc8fa36097c49c88c1b5b981a32e2a2 P e4cfb8fc5a474177af6848a6efa13a27: MemRowSet was empty: no flush needed.
I20240502 14:14:51.813331 26797 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:14:51.821463 26407 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:14:51.822800 30001 consensus_queue.cc:1035] T 39e6a71b3eef40848ea1ba4ef9604fa2 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:51.827913 29288 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:51.849582 30007 tablet.cc:1722] T edc8fa36097c49c88c1b5b981a32e2a2 P 89af7f423cf44717a08d6b142f0ed49d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:51.850997 30007 tablet.cc:1620] T edc8fa36097c49c88c1b5b981a32e2a2 P 89af7f423cf44717a08d6b142f0ed49d: MemRowSet was empty: no flush needed.
I20240502 14:14:51.855841 26283 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_extra_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:51.859408 26277 catalog_manager.cc:5576] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985) does not have the latest schema for tablet edc8fa36097c49c88c1b5b981a32e2a2 (table rebalancer_dynamic_table [id=02258aeb7420405ab4cce9f74610170f]). Expected version 1 got 0
I20240502 14:14:51.868712 26285 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240502 14:14:51.869050 26285 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240502 14:14:51.879356 26797 tablet_service.cc:1467] Processing CreateTablet for tablet dc3740d0671d42bc90eb9baeb02f5b0a (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=fe880af4989f416d82ae711576173990]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:51.879364 29555 tablet_service.cc:1467] Processing CreateTablet for tablet 01e6bc3689db46e2ad7d0484a29af287 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=fe880af4989f416d82ae711576173990]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:51.879917 26797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc3740d0671d42bc90eb9baeb02f5b0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:51.879982 29555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01e6bc3689db46e2ad7d0484a29af287. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:51.880754 26285 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet 1a147a6c2ba04d848a122021fd7c63c6
I20240502 14:14:51.881302 26285 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet e79bc37363cc48e7943e0275981be31a
I20240502 14:14:51.881702 26285 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet 2924b2da6f8347aca55042bfbf741c38
I20240502 14:14:51.883397 29553 tablet_service.cc:1467] Processing CreateTablet for tablet 2532ebd7bdf14876b74a26afd674a7ac (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=fe880af4989f416d82ae711576173990]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:51.883996 29553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2532ebd7bdf14876b74a26afd674a7ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:51.886061 29156 tablet_service.cc:1467] Processing CreateTablet for tablet 2532ebd7bdf14876b74a26afd674a7ac (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=fe880af4989f416d82ae711576173990]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:51.886196 29155 tablet_service.cc:1467] Processing CreateTablet for tablet 01e6bc3689db46e2ad7d0484a29af287 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=fe880af4989f416d82ae711576173990]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:51.886559 29156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2532ebd7bdf14876b74a26afd674a7ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:51.886762 29155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01e6bc3689db46e2ad7d0484a29af287. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:51.897706 29423 tablet_service.cc:1467] Processing CreateTablet for tablet 01e6bc3689db46e2ad7d0484a29af287 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=fe880af4989f416d82ae711576173990]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:51.898286 29423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01e6bc3689db46e2ad7d0484a29af287. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:51.900853 29421 tablet_service.cc:1467] Processing CreateTablet for tablet 2532ebd7bdf14876b74a26afd674a7ac (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=fe880af4989f416d82ae711576173990]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:51.901379 29421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2532ebd7bdf14876b74a26afd674a7ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:51.904533 30004 tablet.cc:1722] T edc8fa36097c49c88c1b5b981a32e2a2 P 1cb67158133e4648a6eceba32d4dbba3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:51.905004 29422 tablet_service.cc:1467] Processing CreateTablet for tablet dc3740d0671d42bc90eb9baeb02f5b0a (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=fe880af4989f416d82ae711576173990]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:51.905227 30014 tablet_bootstrap.cc:492] T 2532ebd7bdf14876b74a26afd674a7ac P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:14:51.905557 29422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc3740d0671d42bc90eb9baeb02f5b0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:51.907052 30014 tablet_bootstrap.cc:654] T 2532ebd7bdf14876b74a26afd674a7ac P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:51.909175 29554 tablet_service.cc:1467] Processing CreateTablet for tablet dc3740d0671d42bc90eb9baeb02f5b0a (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=fe880af4989f416d82ae711576173990]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:51.909802 29554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc3740d0671d42bc90eb9baeb02f5b0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:51.914085 26407 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a147a6c2ba04d848a122021fd7c63c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:51 UTC) from {username='slave'} at 127.0.0.1:37422
I20240502 14:14:51.914523 26406 tablet_service.cc:1514] Processing DeleteTablet for tablet e79bc37363cc48e7943e0275981be31a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:51 UTC) from {username='slave'} at 127.0.0.1:37422
I20240502 14:14:51.914630 26538 tablet_service.cc:1514] Processing DeleteTablet for tablet e79bc37363cc48e7943e0275981be31a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:51 UTC) from {username='slave'} at 127.0.0.1:51696
I20240502 14:14:51.914768 26537 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a147a6c2ba04d848a122021fd7c63c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:51 UTC) from {username='slave'} at 127.0.0.1:51696
I20240502 14:14:51.914812 26536 tablet_service.cc:1514] Processing DeleteTablet for tablet 2924b2da6f8347aca55042bfbf741c38 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:51 UTC) from {username='slave'} at 127.0.0.1:51696
I20240502 14:14:51.918180 26797 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a147a6c2ba04d848a122021fd7c63c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:51 UTC) from {username='slave'} at 127.0.0.1:56010
I20240502 14:14:51.918247 26798 tablet_service.cc:1514] Processing DeleteTablet for tablet 2924b2da6f8347aca55042bfbf741c38 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:51 UTC) from {username='slave'} at 127.0.0.1:56010
I20240502 14:14:51.918983 30004 tablet.cc:1620] T edc8fa36097c49c88c1b5b981a32e2a2 P 1cb67158133e4648a6eceba32d4dbba3: MemRowSet was empty: no flush needed.
I20240502 14:14:51.919953 30015 tablet_bootstrap.cc:492] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:51.922119 26668 tablet_service.cc:1514] Processing DeleteTablet for tablet e79bc37363cc48e7943e0275981be31a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:51 UTC) from {username='slave'} at 127.0.0.1:34150
I20240502 14:14:51.922616 26669 tablet_service.cc:1514] Processing DeleteTablet for tablet 2924b2da6f8347aca55042bfbf741c38 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:51 UTC) from {username='slave'} at 127.0.0.1:34150
I20240502 14:14:51.929456 30015 tablet_bootstrap.cc:654] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:51.938419 30018 tablet_replica.cc:330] T e79bc37363cc48e7943e0275981be31a P ba187e38a2754d60a12de9ac4856430a: stopping tablet replica
I20240502 14:14:51.939078 30018 raft_consensus.cc:2229] T e79bc37363cc48e7943e0275981be31a P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:51.939778 30018 raft_consensus.cc:2258] T e79bc37363cc48e7943e0275981be31a P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:51.940994 30018 ts_tablet_manager.cc:1875] T e79bc37363cc48e7943e0275981be31a P ba187e38a2754d60a12de9ac4856430a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:51.943559 30014 tablet_bootstrap.cc:492] T 2532ebd7bdf14876b74a26afd674a7ac P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:14:51.943835 30014 ts_tablet_manager.cc:1367] T 2532ebd7bdf14876b74a26afd674a7ac P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.000s
I20240502 14:14:51.946662 30015 tablet_bootstrap.cc:492] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:51.947537 30014 raft_consensus.cc:348] T 2532ebd7bdf14876b74a26afd674a7ac P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:51.962870 30014 raft_consensus.cc:374] T 2532ebd7bdf14876b74a26afd674a7ac P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:51.963557 30015 ts_tablet_manager.cc:1367] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.052s user 0.004s sys 0.000s
I20240502 14:14:51.963887 30014 raft_consensus.cc:729] T 2532ebd7bdf14876b74a26afd674a7ac P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:14:51.965585 30015 raft_consensus.cc:348] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:51.966586 30014 consensus_queue.cc:260] T 2532ebd7bdf14876b74a26afd674a7ac P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:51.967562 30020 tablet_replica.cc:330] T e79bc37363cc48e7943e0275981be31a P 69ab78b2a1b74d2bbf2cdeafa0a4af23: stopping tablet replica
I20240502 14:14:51.967939 30020 raft_consensus.cc:2229] T e79bc37363cc48e7943e0275981be31a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:51.968083 30014 ts_tablet_manager.cc:1398] T 2532ebd7bdf14876b74a26afd674a7ac P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20240502 14:14:51.968477 30020 raft_consensus.cc:2258] T e79bc37363cc48e7943e0275981be31a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:51.970803 30020 ts_tablet_manager.cc:1875] T e79bc37363cc48e7943e0275981be31a P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:51.972276 30017 tablet_replica.cc:330] T 1a147a6c2ba04d848a122021fd7c63c6 P 89af7f423cf44717a08d6b142f0ed49d: stopping tablet replica
I20240502 14:14:51.972714 30017 raft_consensus.cc:2229] T 1a147a6c2ba04d848a122021fd7c63c6 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:51.973435 30017 raft_consensus.cc:2258] T 1a147a6c2ba04d848a122021fd7c63c6 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:51.974529 30017 ts_tablet_manager.cc:1875] T 1a147a6c2ba04d848a122021fd7c63c6 P 89af7f423cf44717a08d6b142f0ed49d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:51.976800 30014 tablet_bootstrap.cc:492] T 01e6bc3689db46e2ad7d0484a29af287 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:14:51.978533 30022 tablet_bootstrap.cc:492] T dc3740d0671d42bc90eb9baeb02f5b0a P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:14:51.978588 30014 tablet_bootstrap.cc:654] T 01e6bc3689db46e2ad7d0484a29af287 P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:51.979555 30015 raft_consensus.cc:374] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:51.979686 30015 raft_consensus.cc:729] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:51.979977 30015 consensus_queue.cc:260] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:51.980544 30023 tablet_bootstrap.cc:492] T dc3740d0671d42bc90eb9baeb02f5b0a P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:14:51.982892 30015 ts_tablet_manager.cc:1398] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20240502 14:14:51.983924 30022 tablet_bootstrap.cc:654] T dc3740d0671d42bc90eb9baeb02f5b0a P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:51.984807 30023 tablet_bootstrap.cc:654] T dc3740d0671d42bc90eb9baeb02f5b0a P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:51.993077 30022 tablet_bootstrap.cc:492] T dc3740d0671d42bc90eb9baeb02f5b0a P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:14:51.994297 30014 tablet_bootstrap.cc:492] T 01e6bc3689db46e2ad7d0484a29af287 P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:14:51.994557 30014 ts_tablet_manager.cc:1367] T 01e6bc3689db46e2ad7d0484a29af287 P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20240502 14:14:51.995541 30014 raft_consensus.cc:348] T 01e6bc3689db46e2ad7d0484a29af287 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:51.995944 30022 ts_tablet_manager.cc:1367] T dc3740d0671d42bc90eb9baeb02f5b0a P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20240502 14:14:51.976415 30019 tablet_replica.cc:330] T 1a147a6c2ba04d848a122021fd7c63c6 P e4cfb8fc5a474177af6848a6efa13a27: stopping tablet replica
I20240502 14:14:51.997787 30014 raft_consensus.cc:374] T 01e6bc3689db46e2ad7d0484a29af287 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:51.997890 30019 raft_consensus.cc:2229] T 1a147a6c2ba04d848a122021fd7c63c6 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:51.997947 30014 raft_consensus.cc:729] T 01e6bc3689db46e2ad7d0484a29af287 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:14:51.998333 30014 consensus_queue.cc:260] T 01e6bc3689db46e2ad7d0484a29af287 P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:51.998661 30018 ts_tablet_manager.cc:1888] T e79bc37363cc48e7943e0275981be31a P ba187e38a2754d60a12de9ac4856430a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:51.998853 30019 raft_consensus.cc:2258] T 1a147a6c2ba04d848a122021fd7c63c6 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:51.996913 30022 raft_consensus.cc:348] T dc3740d0671d42bc90eb9baeb02f5b0a P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:51.999718 30019 ts_tablet_manager.cc:1875] T 1a147a6c2ba04d848a122021fd7c63c6 P e4cfb8fc5a474177af6848a6efa13a27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.000427 30014 ts_tablet_manager.cc:1398] T 01e6bc3689db46e2ad7d0484a29af287 P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:14:52.000592 30022 raft_consensus.cc:374] T dc3740d0671d42bc90eb9baeb02f5b0a P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:52.000742 30022 raft_consensus.cc:729] T dc3740d0671d42bc90eb9baeb02f5b0a P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:14:52.000813 30018 log.cc:1196] T e79bc37363cc48e7943e0275981be31a P ba187e38a2754d60a12de9ac4856430a: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/e79bc37363cc48e7943e0275981be31a
I20240502 14:14:52.001075 30022 consensus_queue.cc:260] T dc3740d0671d42bc90eb9baeb02f5b0a P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:52.003510 30022 ts_tablet_manager.cc:1398] T dc3740d0671d42bc90eb9baeb02f5b0a P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:14:52.004427 30022 tablet_bootstrap.cc:492] T 2532ebd7bdf14876b74a26afd674a7ac P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:14:52.007102 30022 tablet_bootstrap.cc:654] T 2532ebd7bdf14876b74a26afd674a7ac P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:52.008289 30023 tablet_bootstrap.cc:492] T dc3740d0671d42bc90eb9baeb02f5b0a P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:14:52.008584 30023 ts_tablet_manager.cc:1367] T dc3740d0671d42bc90eb9baeb02f5b0a P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.000s
I20240502 14:14:52.008860 30018 ts_tablet_manager.cc:1909] T e79bc37363cc48e7943e0275981be31a P ba187e38a2754d60a12de9ac4856430a: Deleting consensus metadata
I20240502 14:14:52.009593 30023 raft_consensus.cc:348] T dc3740d0671d42bc90eb9baeb02f5b0a P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:52.009936 30023 raft_consensus.cc:374] T dc3740d0671d42bc90eb9baeb02f5b0a P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:52.010141 30023 raft_consensus.cc:729] T dc3740d0671d42bc90eb9baeb02f5b0a P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:14:52.010581 30023 consensus_queue.cc:260] T dc3740d0671d42bc90eb9baeb02f5b0a P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:52.010938 30018 tablet_replica.cc:330] T 1a147a6c2ba04d848a122021fd7c63c6 P ba187e38a2754d60a12de9ac4856430a: stopping tablet replica
I20240502 14:14:52.011138 26271 catalog_manager.cc:4830] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): tablet e79bc37363cc48e7943e0275981be31a (table rebalance_test_table_12 [id=52458461122d4161bd91b41ead7538e2]) successfully deleted
I20240502 14:14:52.012537 30023 ts_tablet_manager.cc:1398] T dc3740d0671d42bc90eb9baeb02f5b0a P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:14:52.012897 30023 tablet_bootstrap.cc:492] T 2532ebd7bdf14876b74a26afd674a7ac P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:14:52.015133 30023 tablet_bootstrap.cc:654] T 2532ebd7bdf14876b74a26afd674a7ac P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:52.018066 30022 tablet_bootstrap.cc:492] T 2532ebd7bdf14876b74a26afd674a7ac P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:14:52.018360 30022 ts_tablet_manager.cc:1367] T 2532ebd7bdf14876b74a26afd674a7ac P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240502 14:14:52.019429 30022 raft_consensus.cc:348] T 2532ebd7bdf14876b74a26afd674a7ac P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:52.019771 30022 raft_consensus.cc:374] T 2532ebd7bdf14876b74a26afd674a7ac P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:52.019896 30022 raft_consensus.cc:729] T 2532ebd7bdf14876b74a26afd674a7ac P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:14:52.020260 30022 consensus_queue.cc:260] T 2532ebd7bdf14876b74a26afd674a7ac P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:52.021035 30022 ts_tablet_manager.cc:1398] T 2532ebd7bdf14876b74a26afd674a7ac P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:52.021464 30022 tablet_bootstrap.cc:492] T 01e6bc3689db46e2ad7d0484a29af287 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:14:52.023280 30022 tablet_bootstrap.cc:654] T 01e6bc3689db46e2ad7d0484a29af287 P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:52.024727 30018 raft_consensus.cc:2229] T 1a147a6c2ba04d848a122021fd7c63c6 P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:52.028877 30018 raft_consensus.cc:2258] T 1a147a6c2ba04d848a122021fd7c63c6 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.030247 26279 catalog_manager.cc:5576] TS 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715) does not have the latest schema for tablet edc8fa36097c49c88c1b5b981a32e2a2 (table rebalancer_dynamic_table [id=02258aeb7420405ab4cce9f74610170f]). Expected version 1 got 0
I20240502 14:14:52.033216 30018 ts_tablet_manager.cc:1875] T 1a147a6c2ba04d848a122021fd7c63c6 P ba187e38a2754d60a12de9ac4856430a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.034658 30023 tablet_bootstrap.cc:492] T 2532ebd7bdf14876b74a26afd674a7ac P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:14:52.034866 30023 ts_tablet_manager.cc:1367] T 2532ebd7bdf14876b74a26afd674a7ac P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20240502 14:14:52.035750 30023 raft_consensus.cc:348] T 2532ebd7bdf14876b74a26afd674a7ac P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:52.035996 30023 raft_consensus.cc:374] T 2532ebd7bdf14876b74a26afd674a7ac P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:52.036082 30023 raft_consensus.cc:729] T 2532ebd7bdf14876b74a26afd674a7ac P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:14:52.036311 30023 consensus_queue.cc:260] T 2532ebd7bdf14876b74a26afd674a7ac P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:52.036916 30023 ts_tablet_manager.cc:1398] T 2532ebd7bdf14876b74a26afd674a7ac P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:52.037279 30023 tablet_bootstrap.cc:492] T 01e6bc3689db46e2ad7d0484a29af287 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:14:52.039124 30023 tablet_bootstrap.cc:654] T 01e6bc3689db46e2ad7d0484a29af287 P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:52.045780 29226 raft_consensus.cc:482] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:52.046002 29226 raft_consensus.cc:504] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:52.047063 29226 leader_election.cc:288] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:14:52.051865 30022 tablet_bootstrap.cc:492] T 01e6bc3689db46e2ad7d0484a29af287 P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:14:52.052896 30022 ts_tablet_manager.cc:1367] T 01e6bc3689db46e2ad7d0484a29af287 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.011s
I20240502 14:14:52.054082 30022 raft_consensus.cc:348] T 01e6bc3689db46e2ad7d0484a29af287 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:52.055773 30022 raft_consensus.cc:374] T 01e6bc3689db46e2ad7d0484a29af287 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:52.056068 30022 raft_consensus.cc:729] T 01e6bc3689db46e2ad7d0484a29af287 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:14:52.056695 30022 consensus_queue.cc:260] T 01e6bc3689db46e2ad7d0484a29af287 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:52.059584 30022 ts_tablet_manager.cc:1398] T 01e6bc3689db46e2ad7d0484a29af287 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 14:14:52.065310 26283 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=02258aeb7420405ab4cce9f74610170f] alter complete (version 1)
I20240502 14:14:52.072757 30018 ts_tablet_manager.cc:1888] T 1a147a6c2ba04d848a122021fd7c63c6 P ba187e38a2754d60a12de9ac4856430a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:52.073112 30018 log.cc:1196] T 1a147a6c2ba04d848a122021fd7c63c6 P ba187e38a2754d60a12de9ac4856430a: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/1a147a6c2ba04d848a122021fd7c63c6
I20240502 14:14:52.076671 30018 ts_tablet_manager.cc:1909] T 1a147a6c2ba04d848a122021fd7c63c6 P ba187e38a2754d60a12de9ac4856430a: Deleting consensus metadata
I20240502 14:14:52.077937 26271 catalog_manager.cc:4830] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): tablet 1a147a6c2ba04d848a122021fd7c63c6 (table rebalance_test_table_12 [id=52458461122d4161bd91b41ead7538e2]) successfully deleted
I20240502 14:14:52.078152 30018 tablet_replica.cc:330] T 2924b2da6f8347aca55042bfbf741c38 P ba187e38a2754d60a12de9ac4856430a: stopping tablet replica
I20240502 14:14:52.078377 30018 raft_consensus.cc:2229] T 2924b2da6f8347aca55042bfbf741c38 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:52.080202 30023 tablet_bootstrap.cc:492] T 01e6bc3689db46e2ad7d0484a29af287 P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:14:52.080502 30023 ts_tablet_manager.cc:1367] T 01e6bc3689db46e2ad7d0484a29af287 P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.043s user 0.010s sys 0.017s
I20240502 14:14:52.080974 29442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc3740d0671d42bc90eb9baeb02f5b0a" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:14:52.081244 29442 raft_consensus.cc:2454] T dc3740d0671d42bc90eb9baeb02f5b0a P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 0.
I20240502 14:14:52.081246 29575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc3740d0671d42bc90eb9baeb02f5b0a" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86" is_pre_election: true
I20240502 14:14:52.081468 29575 raft_consensus.cc:2454] T dc3740d0671d42bc90eb9baeb02f5b0a P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 0.
I20240502 14:14:52.081533 30023 raft_consensus.cc:348] T 01e6bc3689db46e2ad7d0484a29af287 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:52.081841 30023 raft_consensus.cc:374] T 01e6bc3689db46e2ad7d0484a29af287 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:52.081827 26755 leader_election.cc:302] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:52.082069 30018 raft_consensus.cc:2258] T 2924b2da6f8347aca55042bfbf741c38 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.082068 30023 raft_consensus.cc:729] T 01e6bc3689db46e2ad7d0484a29af287 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:14:52.082142 29226 raft_consensus.cc:2781] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:52.082254 29226 raft_consensus.cc:482] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:52.082370 29226 raft_consensus.cc:3037] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:52.082399 30023 consensus_queue.cc:260] T 01e6bc3689db46e2ad7d0484a29af287 P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:52.083067 29704 raft_consensus.cc:482] T 2532ebd7bdf14876b74a26afd674a7ac P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:52.083214 29704 raft_consensus.cc:504] T 2532ebd7bdf14876b74a26afd674a7ac P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:52.083822 29704 leader_election.cc:288] T 2532ebd7bdf14876b74a26afd674a7ac P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:14:52.084183 30023 ts_tablet_manager.cc:1398] T 01e6bc3689db46e2ad7d0484a29af287 P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:52.084995 29442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2532ebd7bdf14876b74a26afd674a7ac" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:14:52.085238 29442 raft_consensus.cc:2454] T 2532ebd7bdf14876b74a26afd674a7ac P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:14:52.085240 29174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2532ebd7bdf14876b74a26afd674a7ac" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:14:52.085348 29226 raft_consensus.cc:504] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:52.085507 29174 raft_consensus.cc:2454] T 2532ebd7bdf14876b74a26afd674a7ac P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:14:52.085876 29511 leader_election.cc:302] T 2532ebd7bdf14876b74a26afd674a7ac P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, 68a3c2b17ed94e06aba66ed4178132ab; no voters:
I20240502 14:14:52.086107 29226 leader_election.cc:288] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Requested vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:14:52.086318 29704 raft_consensus.cc:2781] T 2532ebd7bdf14876b74a26afd674a7ac P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:52.086444 29704 raft_consensus.cc:482] T 2532ebd7bdf14876b74a26afd674a7ac P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:52.086529 29704 raft_consensus.cc:3037] T 2532ebd7bdf14876b74a26afd674a7ac P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:52.087119 29442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc3740d0671d42bc90eb9baeb02f5b0a" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
I20240502 14:14:52.087316 29442 raft_consensus.cc:3037] T dc3740d0671d42bc90eb9baeb02f5b0a P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:52.087924 30018 ts_tablet_manager.cc:1875] T 2924b2da6f8347aca55042bfbf741c38 P ba187e38a2754d60a12de9ac4856430a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.089607 29442 raft_consensus.cc:2454] T dc3740d0671d42bc90eb9baeb02f5b0a P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
I20240502 14:14:52.089668 29704 raft_consensus.cc:504] T 2532ebd7bdf14876b74a26afd674a7ac P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:52.090669 29575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc3740d0671d42bc90eb9baeb02f5b0a" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86"
I20240502 14:14:52.090884 29575 raft_consensus.cc:3037] T dc3740d0671d42bc90eb9baeb02f5b0a P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:52.091630 29174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2532ebd7bdf14876b74a26afd674a7ac" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:14:52.091862 29174 raft_consensus.cc:3037] T 2532ebd7bdf14876b74a26afd674a7ac P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:52.093848 29575 raft_consensus.cc:2454] T dc3740d0671d42bc90eb9baeb02f5b0a P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
I20240502 14:14:52.094257 26755 leader_election.cc:302] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:52.094779 29174 raft_consensus.cc:2454] T 2532ebd7bdf14876b74a26afd674a7ac P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:14:52.095801 29510 leader_election.cc:302] T 2532ebd7bdf14876b74a26afd674a7ac P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 4092902b3e674857a0147f1b4f2cdb86; no voters:
I20240502 14:14:52.096114 29694 raft_consensus.cc:2781] T 2532ebd7bdf14876b74a26afd674a7ac P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:52.098502 29704 leader_election.cc:288] T 2532ebd7bdf14876b74a26afd674a7ac P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Requested vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:14:52.103335 29943 raft_consensus.cc:2781] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:52.103502 29943 raft_consensus.cc:686] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Becoming Leader. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Running, Role: LEADER
I20240502 14:14:52.103797 29943 consensus_queue.cc:237] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:14:52.090783 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2532ebd7bdf14876b74a26afd674a7ac" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
I20240502 14:14:52.105772 29443 raft_consensus.cc:3037] T 2532ebd7bdf14876b74a26afd674a7ac P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:52.105247 30008 raft_consensus.cc:482] T 01e6bc3689db46e2ad7d0484a29af287 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:52.108539 30008 raft_consensus.cc:504] T 01e6bc3689db46e2ad7d0484a29af287 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:52.108701 30017 ts_tablet_manager.cc:1888] T 1a147a6c2ba04d848a122021fd7c63c6 P 89af7f423cf44717a08d6b142f0ed49d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:52.108842 30017 log.cc:1196] T 1a147a6c2ba04d848a122021fd7c63c6 P 89af7f423cf44717a08d6b142f0ed49d: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/1a147a6c2ba04d848a122021fd7c63c6
I20240502 14:14:52.109181 29694 raft_consensus.cc:686] T 2532ebd7bdf14876b74a26afd674a7ac P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Becoming Leader. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Running, Role: LEADER
I20240502 14:14:52.109625 30008 leader_election.cc:288] T 01e6bc3689db46e2ad7d0484a29af287 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:14:52.109532 29694 consensus_queue.cc:237] T 2532ebd7bdf14876b74a26afd674a7ac P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:52.110801 29174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01e6bc3689db46e2ad7d0484a29af287" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:14:52.111039 29174 raft_consensus.cc:2454] T 01e6bc3689db46e2ad7d0484a29af287 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 0.
I20240502 14:14:52.111892 29378 leader_election.cc:302] T 01e6bc3689db46e2ad7d0484a29af287 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 68a3c2b17ed94e06aba66ed4178132ab; no voters:
I20240502 14:14:52.114091 30008 raft_consensus.cc:2781] T 01e6bc3689db46e2ad7d0484a29af287 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:52.114269 30008 raft_consensus.cc:482] T 01e6bc3689db46e2ad7d0484a29af287 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:52.114384 30008 raft_consensus.cc:3037] T 01e6bc3689db46e2ad7d0484a29af287 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:52.118736 29443 raft_consensus.cc:2454] T 2532ebd7bdf14876b74a26afd674a7ac P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:14:52.122721 30008 raft_consensus.cc:504] T 01e6bc3689db46e2ad7d0484a29af287 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:52.124169 30019 ts_tablet_manager.cc:1888] T 1a147a6c2ba04d848a122021fd7c63c6 P e4cfb8fc5a474177af6848a6efa13a27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:52.125164 30019 log.cc:1196] T 1a147a6c2ba04d848a122021fd7c63c6 P e4cfb8fc5a474177af6848a6efa13a27: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/wals/1a147a6c2ba04d848a122021fd7c63c6
I20240502 14:14:52.125936 30019 ts_tablet_manager.cc:1909] T 1a147a6c2ba04d848a122021fd7c63c6 P e4cfb8fc5a474177af6848a6efa13a27: Deleting consensus metadata
I20240502 14:14:52.126591 30020 ts_tablet_manager.cc:1888] T e79bc37363cc48e7943e0275981be31a P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:52.109371 30017 ts_tablet_manager.cc:1909] T 1a147a6c2ba04d848a122021fd7c63c6 P 89af7f423cf44717a08d6b142f0ed49d: Deleting consensus metadata
I20240502 14:14:52.128541 29174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01e6bc3689db46e2ad7d0484a29af287" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:14:52.128791 29174 raft_consensus.cc:3037] T 01e6bc3689db46e2ad7d0484a29af287 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:52.129957 30019 tablet_replica.cc:330] T 2924b2da6f8347aca55042bfbf741c38 P e4cfb8fc5a474177af6848a6efa13a27: stopping tablet replica
I20240502 14:14:52.131042 30017 tablet_replica.cc:330] T e79bc37363cc48e7943e0275981be31a P 89af7f423cf44717a08d6b142f0ed49d: stopping tablet replica
I20240502 14:14:52.131323 30020 log.cc:1196] T e79bc37363cc48e7943e0275981be31a P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/wals/e79bc37363cc48e7943e0275981be31a
I20240502 14:14:52.131722 29174 raft_consensus.cc:2454] T 01e6bc3689db46e2ad7d0484a29af287 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 1.
I20240502 14:14:52.132266 26271 catalog_manager.cc:4830] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): tablet 1a147a6c2ba04d848a122021fd7c63c6 (table rebalance_test_table_12 [id=52458461122d4161bd91b41ead7538e2]) successfully deleted
I20240502 14:14:52.132467 30020 ts_tablet_manager.cc:1909] T e79bc37363cc48e7943e0275981be31a P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting consensus metadata
I20240502 14:14:52.132524 29378 leader_election.cc:302] T 01e6bc3689db46e2ad7d0484a29af287 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 68a3c2b17ed94e06aba66ed4178132ab; no voters:
I20240502 14:14:52.132913 29652 raft_consensus.cc:2781] T 01e6bc3689db46e2ad7d0484a29af287 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:52.133570 30020 tablet_replica.cc:330] T 2924b2da6f8347aca55042bfbf741c38 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: stopping tablet replica
I20240502 14:14:52.133980 26272 catalog_manager.cc:4830] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): tablet e79bc37363cc48e7943e0275981be31a (table rebalance_test_table_12 [id=52458461122d4161bd91b41ead7538e2]) successfully deleted
I20240502 14:14:52.134546 30017 raft_consensus.cc:2229] T e79bc37363cc48e7943e0275981be31a P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:52.135071 30019 raft_consensus.cc:2229] T 2924b2da6f8347aca55042bfbf741c38 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:52.136705 30017 raft_consensus.cc:2258] T e79bc37363cc48e7943e0275981be31a P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.137202 30019 raft_consensus.cc:2258] T 2924b2da6f8347aca55042bfbf741c38 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.138530 30017 ts_tablet_manager.cc:1875] T e79bc37363cc48e7943e0275981be31a P 89af7f423cf44717a08d6b142f0ed49d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.139668 30019 ts_tablet_manager.cc:1875] T 2924b2da6f8347aca55042bfbf741c38 P e4cfb8fc5a474177af6848a6efa13a27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.140233 30020 raft_consensus.cc:2229] T 2924b2da6f8347aca55042bfbf741c38 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:52.140599 26272 catalog_manager.cc:4830] TS e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): tablet 1a147a6c2ba04d848a122021fd7c63c6 (table rebalance_test_table_12 [id=52458461122d4161bd91b41ead7538e2]) successfully deleted
I20240502 14:14:52.142185 30008 leader_election.cc:288] T 01e6bc3689db46e2ad7d0484a29af287 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 election: Requested vote from peers 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:14:52.144819 30020 raft_consensus.cc:2258] T 2924b2da6f8347aca55042bfbf741c38 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.149408 30020 ts_tablet_manager.cc:1875] T 2924b2da6f8347aca55042bfbf741c38 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.152041 30018 ts_tablet_manager.cc:1888] T 2924b2da6f8347aca55042bfbf741c38 P ba187e38a2754d60a12de9ac4856430a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:52.157780 30018 log.cc:1196] T 2924b2da6f8347aca55042bfbf741c38 P ba187e38a2754d60a12de9ac4856430a: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/2924b2da6f8347aca55042bfbf741c38
I20240502 14:14:52.160748 30020 ts_tablet_manager.cc:1888] T 2924b2da6f8347aca55042bfbf741c38 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:52.161008 30020 log.cc:1196] T 2924b2da6f8347aca55042bfbf741c38 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/wals/2924b2da6f8347aca55042bfbf741c38
I20240502 14:14:52.161690 30020 ts_tablet_manager.cc:1909] T 2924b2da6f8347aca55042bfbf741c38 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting consensus metadata
I20240502 14:14:52.163115 26272 catalog_manager.cc:4830] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): tablet 2924b2da6f8347aca55042bfbf741c38 (table rebalance_test_table_12 [id=52458461122d4161bd91b41ead7538e2]) successfully deleted
I20240502 14:14:52.162977 26279 catalog_manager.cc:5484] T 2532ebd7bdf14876b74a26afd674a7ac P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200). New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:14:52.167143 30018 ts_tablet_manager.cc:1909] T 2924b2da6f8347aca55042bfbf741c38 P ba187e38a2754d60a12de9ac4856430a: Deleting consensus metadata
I20240502 14:14:52.168426 26271 catalog_manager.cc:4830] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): tablet 2924b2da6f8347aca55042bfbf741c38 (table rebalance_test_table_12 [id=52458461122d4161bd91b41ead7538e2]) successfully deleted
I20240502 14:14:52.174111 29652 raft_consensus.cc:686] T 01e6bc3689db46e2ad7d0484a29af287 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 LEADER]: Becoming Leader. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Running, Role: LEADER
I20240502 14:14:52.174690 29652 consensus_queue.cc:237] T 01e6bc3689db46e2ad7d0484a29af287 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:52.188709 29575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01e6bc3689db46e2ad7d0484a29af287" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86" is_pre_election: true
I20240502 14:14:52.189023 29575 raft_consensus.cc:2454] T 01e6bc3689db46e2ad7d0484a29af287 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 0.
I20240502 14:14:52.189312 29574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01e6bc3689db46e2ad7d0484a29af287" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86"
I20240502 14:14:52.189477 29574 raft_consensus.cc:3037] T 01e6bc3689db46e2ad7d0484a29af287 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:52.193260 29574 raft_consensus.cc:2454] T 01e6bc3689db46e2ad7d0484a29af287 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 1.
I20240502 14:14:52.195588 30019 ts_tablet_manager.cc:1888] T 2924b2da6f8347aca55042bfbf741c38 P e4cfb8fc5a474177af6848a6efa13a27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:52.196632 30017 ts_tablet_manager.cc:1888] T e79bc37363cc48e7943e0275981be31a P 89af7f423cf44717a08d6b142f0ed49d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:52.196871 30017 log.cc:1196] T e79bc37363cc48e7943e0275981be31a P 89af7f423cf44717a08d6b142f0ed49d: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/e79bc37363cc48e7943e0275981be31a
I20240502 14:14:52.197386 30017 ts_tablet_manager.cc:1909] T e79bc37363cc48e7943e0275981be31a P 89af7f423cf44717a08d6b142f0ed49d: Deleting consensus metadata
I20240502 14:14:52.198293 30019 log.cc:1196] T 2924b2da6f8347aca55042bfbf741c38 P e4cfb8fc5a474177af6848a6efa13a27: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/wals/2924b2da6f8347aca55042bfbf741c38
I20240502 14:14:52.198385 26271 catalog_manager.cc:4830] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): tablet e79bc37363cc48e7943e0275981be31a (table rebalance_test_table_12 [id=52458461122d4161bd91b41ead7538e2]) successfully deleted
I20240502 14:14:52.198827 30019 ts_tablet_manager.cc:1909] T 2924b2da6f8347aca55042bfbf741c38 P e4cfb8fc5a474177af6848a6efa13a27: Deleting consensus metadata
W20240502 14:14:52.199709 26865 tablet.cc:2369] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:14:52.199708 26272 catalog_manager.cc:4830] TS e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): tablet 2924b2da6f8347aca55042bfbf741c38 (table rebalance_test_table_12 [id=52458461122d4161bd91b41ead7538e2]) successfully deleted
I20240502 14:14:52.197328 26279 catalog_manager.cc:5484] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196). New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:14:52.207906 26283 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
W20240502 14:14:52.229995 26363 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 48395.9us
I20240502 14:14:52.253834 26283 catalog_manager.cc:5484] T 01e6bc3689db46e2ad7d0484a29af287 P 68a3c2b17ed94e06aba66ed4178132ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199). New cstate: current_term: 1 leader_uuid: "68a3c2b17ed94e06aba66ed4178132ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:14:52.255053 29995 tablet.cc:1722] T edc8fa36097c49c88c1b5b981a32e2a2 P e4cfb8fc5a474177af6848a6efa13a27: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:52.255618 29995 tablet.cc:1620] T edc8fa36097c49c88c1b5b981a32e2a2 P e4cfb8fc5a474177af6848a6efa13a27: MemRowSet was empty: no flush needed.
I20240502 14:14:52.259290 30007 tablet.cc:1722] T edc8fa36097c49c88c1b5b981a32e2a2 P 89af7f423cf44717a08d6b142f0ed49d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:52.259850 30007 tablet.cc:1620] T edc8fa36097c49c88c1b5b981a32e2a2 P 89af7f423cf44717a08d6b142f0ed49d: MemRowSet was empty: no flush needed.
I20240502 14:14:52.260461 30004 tablet.cc:1722] T edc8fa36097c49c88c1b5b981a32e2a2 P 1cb67158133e4648a6eceba32d4dbba3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:52.260962 30004 tablet.cc:1620] T edc8fa36097c49c88c1b5b981a32e2a2 P 1cb67158133e4648a6eceba32d4dbba3: MemRowSet was empty: no flush needed.
I20240502 14:14:52.270859 26279 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=02258aeb7420405ab4cce9f74610170f] alter complete (version 2)
I20240502 14:14:52.373832 26279 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:14:52.378454 26423 consensus_queue.cc:237] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:52.382274 26689 raft_consensus.cc:1261] T 0fc2e6c74dc34a45ae1ec46608642880 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:52.382642 26558 raft_consensus.cc:1261] T 0fc2e6c74dc34a45ae1ec46608642880 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:52.382970 30041 consensus_queue.cc:1035] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:52.384253 26279 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240502 14:14:52.384380 30041 consensus_queue.cc:1035] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:52.384538 26279 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240502 14:14:52.387144 29995 tablet.cc:1722] T edc8fa36097c49c88c1b5b981a32e2a2 P e4cfb8fc5a474177af6848a6efa13a27: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:52.388813 30004 tablet.cc:1722] T edc8fa36097c49c88c1b5b981a32e2a2 P 1cb67158133e4648a6eceba32d4dbba3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:52.396108 26279 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet 6f3334b24a3145e0b5498d8d97eabb1f
I20240502 14:14:52.396744 26279 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet f67582dfce354a9b92078ab0b5799194
I20240502 14:14:52.397274 26279 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet 038a5d3d5368433cac07bdd94a4f779a
I20240502 14:14:52.397648 26536 tablet_service.cc:1514] Processing DeleteTablet for tablet f67582dfce354a9b92078ab0b5799194 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:52 UTC) from {username='slave'} at 127.0.0.1:51696
I20240502 14:14:52.397862 26538 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f3334b24a3145e0b5498d8d97eabb1f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:52 UTC) from {username='slave'} at 127.0.0.1:51696
I20240502 14:14:52.397962 30018 tablet_replica.cc:330] T f67582dfce354a9b92078ab0b5799194 P ba187e38a2754d60a12de9ac4856430a: stopping tablet replica
I20240502 14:14:52.398216 30018 raft_consensus.cc:2229] T f67582dfce354a9b92078ab0b5799194 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:52.398470 26538 tablet_service.cc:1514] Processing DeleteTablet for tablet 038a5d3d5368433cac07bdd94a4f779a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:52 UTC) from {username='slave'} at 127.0.0.1:51696
I20240502 14:14:52.398701 26797 tablet_service.cc:1514] Processing DeleteTablet for tablet f67582dfce354a9b92078ab0b5799194 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:52 UTC) from {username='slave'} at 127.0.0.1:56010
I20240502 14:14:52.398705 26407 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f3334b24a3145e0b5498d8d97eabb1f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:52 UTC) from {username='slave'} at 127.0.0.1:37422
I20240502 14:14:52.398986 30019 tablet_replica.cc:330] T f67582dfce354a9b92078ab0b5799194 P e4cfb8fc5a474177af6848a6efa13a27: stopping tablet replica
I20240502 14:14:52.399013 26285 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table_new_name [id=02258aeb7420405ab4cce9f74610170f] alter complete (version 3)
I20240502 14:14:52.399063 26406 tablet_service.cc:1514] Processing DeleteTablet for tablet 038a5d3d5368433cac07bdd94a4f779a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:52 UTC) from {username='slave'} at 127.0.0.1:37422
I20240502 14:14:52.399199 30019 raft_consensus.cc:2229] T f67582dfce354a9b92078ab0b5799194 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:52.399592 30019 raft_consensus.cc:2258] T f67582dfce354a9b92078ab0b5799194 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.400339 26668 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f3334b24a3145e0b5498d8d97eabb1f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:52 UTC) from {username='slave'} at 127.0.0.1:34150
I20240502 14:14:52.400383 30019 ts_tablet_manager.cc:1875] T f67582dfce354a9b92078ab0b5799194 P e4cfb8fc5a474177af6848a6efa13a27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.400666 26669 tablet_service.cc:1514] Processing DeleteTablet for tablet f67582dfce354a9b92078ab0b5799194 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:52 UTC) from {username='slave'} at 127.0.0.1:34150
I20240502 14:14:52.400924 30020 tablet_replica.cc:330] T 6f3334b24a3145e0b5498d8d97eabb1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: stopping tablet replica
I20240502 14:14:52.401130 30020 raft_consensus.cc:2229] T 6f3334b24a3145e0b5498d8d97eabb1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:52.401376 30020 raft_consensus.cc:2258] T 6f3334b24a3145e0b5498d8d97eabb1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.402056 30020 ts_tablet_manager.cc:1875] T 6f3334b24a3145e0b5498d8d97eabb1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.404587 30017 tablet_replica.cc:330] T 6f3334b24a3145e0b5498d8d97eabb1f P 89af7f423cf44717a08d6b142f0ed49d: stopping tablet replica
I20240502 14:14:52.404783 30017 raft_consensus.cc:2229] T 6f3334b24a3145e0b5498d8d97eabb1f P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:52.405079 30017 raft_consensus.cc:2258] T 6f3334b24a3145e0b5498d8d97eabb1f P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.405763 30017 ts_tablet_manager.cc:1875] T 6f3334b24a3145e0b5498d8d97eabb1f P 89af7f423cf44717a08d6b142f0ed49d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.408023 30020 ts_tablet_manager.cc:1888] T 6f3334b24a3145e0b5498d8d97eabb1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:52.408141 30020 log.cc:1196] T 6f3334b24a3145e0b5498d8d97eabb1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/wals/6f3334b24a3145e0b5498d8d97eabb1f
I20240502 14:14:52.408586 30020 ts_tablet_manager.cc:1909] T 6f3334b24a3145e0b5498d8d97eabb1f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting consensus metadata
I20240502 14:14:52.408562 30007 tablet.cc:1722] T edc8fa36097c49c88c1b5b981a32e2a2 P 89af7f423cf44717a08d6b142f0ed49d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:52.408679 26667 tablet_service.cc:1514] Processing DeleteTablet for tablet 038a5d3d5368433cac07bdd94a4f779a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:52 UTC) from {username='slave'} at 127.0.0.1:34150
I20240502 14:14:52.409762 26272 catalog_manager.cc:4830] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): tablet 6f3334b24a3145e0b5498d8d97eabb1f (table rebalance_test_table_14 [id=4b7658f1ae4f4326bedadc9a5b8d88c2]) successfully deleted
I20240502 14:14:52.409844 30020 tablet_replica.cc:330] T f67582dfce354a9b92078ab0b5799194 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: stopping tablet replica
I20240502 14:14:52.410080 30020 raft_consensus.cc:2229] T f67582dfce354a9b92078ab0b5799194 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:52.410383 30020 raft_consensus.cc:2258] T f67582dfce354a9b92078ab0b5799194 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.398504 30018 raft_consensus.cc:2258] T f67582dfce354a9b92078ab0b5799194 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.411231 30020 ts_tablet_manager.cc:1875] T f67582dfce354a9b92078ab0b5799194 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.411808 30018 ts_tablet_manager.cc:1875] T f67582dfce354a9b92078ab0b5799194 P ba187e38a2754d60a12de9ac4856430a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.415261 30017 ts_tablet_manager.cc:1888] T 6f3334b24a3145e0b5498d8d97eabb1f P 89af7f423cf44717a08d6b142f0ed49d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:52.415517 30017 log.cc:1196] T 6f3334b24a3145e0b5498d8d97eabb1f P 89af7f423cf44717a08d6b142f0ed49d: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/6f3334b24a3145e0b5498d8d97eabb1f
I20240502 14:14:52.416015 30017 ts_tablet_manager.cc:1909] T 6f3334b24a3145e0b5498d8d97eabb1f P 89af7f423cf44717a08d6b142f0ed49d: Deleting consensus metadata
I20240502 14:14:52.417025 30017 tablet_replica.cc:330] T 038a5d3d5368433cac07bdd94a4f779a P 89af7f423cf44717a08d6b142f0ed49d: stopping tablet replica
I20240502 14:14:52.417323 30017 raft_consensus.cc:2229] T 038a5d3d5368433cac07bdd94a4f779a P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:52.417749 30017 raft_consensus.cc:2258] T 038a5d3d5368433cac07bdd94a4f779a P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.417816 26271 catalog_manager.cc:4830] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): tablet 6f3334b24a3145e0b5498d8d97eabb1f (table rebalance_test_table_14 [id=4b7658f1ae4f4326bedadc9a5b8d88c2]) successfully deleted
I20240502 14:14:52.417855 30020 ts_tablet_manager.cc:1888] T f67582dfce354a9b92078ab0b5799194 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:52.417999 30020 log.cc:1196] T f67582dfce354a9b92078ab0b5799194 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/wals/f67582dfce354a9b92078ab0b5799194
I20240502 14:14:52.418561 30020 ts_tablet_manager.cc:1909] T f67582dfce354a9b92078ab0b5799194 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting consensus metadata
I20240502 14:14:52.418684 30017 ts_tablet_manager.cc:1875] T 038a5d3d5368433cac07bdd94a4f779a P 89af7f423cf44717a08d6b142f0ed49d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.419574 30019 ts_tablet_manager.cc:1888] T f67582dfce354a9b92078ab0b5799194 P e4cfb8fc5a474177af6848a6efa13a27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:52.420250 30019 log.cc:1196] T f67582dfce354a9b92078ab0b5799194 P e4cfb8fc5a474177af6848a6efa13a27: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/wals/f67582dfce354a9b92078ab0b5799194
I20240502 14:14:52.420737 30019 ts_tablet_manager.cc:1909] T f67582dfce354a9b92078ab0b5799194 P e4cfb8fc5a474177af6848a6efa13a27: Deleting consensus metadata
I20240502 14:14:52.433389 26272 catalog_manager.cc:4830] TS e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): tablet f67582dfce354a9b92078ab0b5799194 (table rebalance_test_table_14 [id=4b7658f1ae4f4326bedadc9a5b8d88c2]) successfully deleted
I20240502 14:14:52.433729 26272 catalog_manager.cc:4830] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): tablet f67582dfce354a9b92078ab0b5799194 (table rebalance_test_table_14 [id=4b7658f1ae4f4326bedadc9a5b8d88c2]) successfully deleted
I20240502 14:14:52.434787 30018 ts_tablet_manager.cc:1888] T f67582dfce354a9b92078ab0b5799194 P ba187e38a2754d60a12de9ac4856430a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:52.436017 30020 tablet_replica.cc:330] T 038a5d3d5368433cac07bdd94a4f779a P 69ab78b2a1b74d2bbf2cdeafa0a4af23: stopping tablet replica
I20240502 14:14:52.436363 30020 raft_consensus.cc:2229] T 038a5d3d5368433cac07bdd94a4f779a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240502 14:14:52.436259 26493 consensus_peers.cc:485] T 6f3334b24a3145e0b5498d8d97eabb1f P ba187e38a2754d60a12de9ac4856430a -> Peer 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): Couldn't send request to peer 89af7f423cf44717a08d6b142f0ed49d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f3334b24a3145e0b5498d8d97eabb1f. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:52.437434 30018 log.cc:1196] T f67582dfce354a9b92078ab0b5799194 P ba187e38a2754d60a12de9ac4856430a: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/f67582dfce354a9b92078ab0b5799194
I20240502 14:14:52.437942 30018 ts_tablet_manager.cc:1909] T f67582dfce354a9b92078ab0b5799194 P ba187e38a2754d60a12de9ac4856430a: Deleting consensus metadata
I20240502 14:14:52.438572 30017 ts_tablet_manager.cc:1888] T 038a5d3d5368433cac07bdd94a4f779a P 89af7f423cf44717a08d6b142f0ed49d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:52.438848 30017 log.cc:1196] T 038a5d3d5368433cac07bdd94a4f779a P 89af7f423cf44717a08d6b142f0ed49d: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/038a5d3d5368433cac07bdd94a4f779a
I20240502 14:14:52.439163 30018 tablet_replica.cc:330] T 6f3334b24a3145e0b5498d8d97eabb1f P ba187e38a2754d60a12de9ac4856430a: stopping tablet replica
I20240502 14:14:52.439329 26271 catalog_manager.cc:4830] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): tablet f67582dfce354a9b92078ab0b5799194 (table rebalance_test_table_14 [id=4b7658f1ae4f4326bedadc9a5b8d88c2]) successfully deleted
I20240502 14:14:52.439424 30017 ts_tablet_manager.cc:1909] T 038a5d3d5368433cac07bdd94a4f779a P 89af7f423cf44717a08d6b142f0ed49d: Deleting consensus metadata
I20240502 14:14:52.439846 30020 raft_consensus.cc:2258] T 038a5d3d5368433cac07bdd94a4f779a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.441908 30020 ts_tablet_manager.cc:1875] T 038a5d3d5368433cac07bdd94a4f779a P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.443399 30018 raft_consensus.cc:2229] T 6f3334b24a3145e0b5498d8d97eabb1f P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:52.446899 30018 raft_consensus.cc:2258] T 6f3334b24a3145e0b5498d8d97eabb1f P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.452780 26271 catalog_manager.cc:4830] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): tablet 038a5d3d5368433cac07bdd94a4f779a (table rebalance_test_table_14 [id=4b7658f1ae4f4326bedadc9a5b8d88c2]) successfully deleted
I20240502 14:14:52.453074 30020 ts_tablet_manager.cc:1888] T 038a5d3d5368433cac07bdd94a4f779a P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:52.453271 30020 log.cc:1196] T 038a5d3d5368433cac07bdd94a4f779a P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/wals/038a5d3d5368433cac07bdd94a4f779a
I20240502 14:14:52.453902 30020 ts_tablet_manager.cc:1909] T 038a5d3d5368433cac07bdd94a4f779a P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting consensus metadata
I20240502 14:14:52.454471 30018 ts_tablet_manager.cc:1875] T 6f3334b24a3145e0b5498d8d97eabb1f P ba187e38a2754d60a12de9ac4856430a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 14:14:52.458602 26362 consensus_peers.cc:485] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): Couldn't send request to peer 1cb67158133e4648a6eceba32d4dbba3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0fc2e6c74dc34a45ae1ec46608642880. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:52.460067 26272 catalog_manager.cc:4830] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): tablet 038a5d3d5368433cac07bdd94a4f779a (table rebalance_test_table_14 [id=4b7658f1ae4f4326bedadc9a5b8d88c2]) successfully deleted
I20240502 14:14:52.459801 30041 raft_consensus.cc:2932] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:14:52.462548 30018 ts_tablet_manager.cc:1888] T 6f3334b24a3145e0b5498d8d97eabb1f P ba187e38a2754d60a12de9ac4856430a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:52.462684 30018 log.cc:1196] T 6f3334b24a3145e0b5498d8d97eabb1f P ba187e38a2754d60a12de9ac4856430a: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/6f3334b24a3145e0b5498d8d97eabb1f
I20240502 14:14:52.463102 30018 ts_tablet_manager.cc:1909] T 6f3334b24a3145e0b5498d8d97eabb1f P ba187e38a2754d60a12de9ac4856430a: Deleting consensus metadata
I20240502 14:14:52.463990 30018 tablet_replica.cc:330] T 038a5d3d5368433cac07bdd94a4f779a P ba187e38a2754d60a12de9ac4856430a: stopping tablet replica
I20240502 14:14:52.464216 26271 catalog_manager.cc:4830] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): tablet 6f3334b24a3145e0b5498d8d97eabb1f (table rebalance_test_table_14 [id=4b7658f1ae4f4326bedadc9a5b8d88c2]) successfully deleted
I20240502 14:14:52.464313 30018 raft_consensus.cc:2229] T 038a5d3d5368433cac07bdd94a4f779a P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:52.464592 30018 raft_consensus.cc:2258] T 038a5d3d5368433cac07bdd94a4f779a P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.465221 30018 ts_tablet_manager.cc:1875] T 038a5d3d5368433cac07bdd94a4f779a P ba187e38a2754d60a12de9ac4856430a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.465437 26689 raft_consensus.cc:2932] T 0fc2e6c74dc34a45ae1ec46608642880 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:14:52.467882 26558 raft_consensus.cc:2932] T 0fc2e6c74dc34a45ae1ec46608642880 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:14:52.471051 26285 catalog_manager.cc:5484] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: config changed from index -1 to 4, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:14:52.491189 30018 ts_tablet_manager.cc:1888] T 038a5d3d5368433cac07bdd94a4f779a P ba187e38a2754d60a12de9ac4856430a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:52.491812 30018 log.cc:1196] T 038a5d3d5368433cac07bdd94a4f779a P ba187e38a2754d60a12de9ac4856430a: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/038a5d3d5368433cac07bdd94a4f779a
I20240502 14:14:52.492507 30018 ts_tablet_manager.cc:1909] T 038a5d3d5368433cac07bdd94a4f779a P ba187e38a2754d60a12de9ac4856430a: Deleting consensus metadata
I20240502 14:14:52.493974 26271 catalog_manager.cc:4830] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): tablet 038a5d3d5368433cac07bdd94a4f779a (table rebalance_test_table_14 [id=4b7658f1ae4f4326bedadc9a5b8d88c2]) successfully deleted
I20240502 14:14:52.518787 26279 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:14:52.519028 26279 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:14:52.523614 26279 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet edc8fa36097c49c88c1b5b981a32e2a2
I20240502 14:14:52.524514 26797 tablet_service.cc:1514] Processing DeleteTablet for tablet edc8fa36097c49c88c1b5b981a32e2a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:52 UTC) from {username='slave'} at 127.0.0.1:56010
I20240502 14:14:52.524608 29155 tablet_service.cc:1514] Processing DeleteTablet for tablet edc8fa36097c49c88c1b5b981a32e2a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:52 UTC) from {username='slave'} at 127.0.0.1:55202
I20240502 14:14:52.524904 30019 tablet_replica.cc:330] T edc8fa36097c49c88c1b5b981a32e2a2 P e4cfb8fc5a474177af6848a6efa13a27: stopping tablet replica
I20240502 14:14:52.525199 30019 raft_consensus.cc:2229] T edc8fa36097c49c88c1b5b981a32e2a2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:52.525624 30019 raft_consensus.cc:2258] T edc8fa36097c49c88c1b5b981a32e2a2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.525969 26406 tablet_service.cc:1514] Processing DeleteTablet for tablet edc8fa36097c49c88c1b5b981a32e2a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:52 UTC) from {username='slave'} at 127.0.0.1:37422
I20240502 14:14:52.526298 30017 tablet_replica.cc:330] T edc8fa36097c49c88c1b5b981a32e2a2 P 89af7f423cf44717a08d6b142f0ed49d: stopping tablet replica
I20240502 14:14:52.526360 30075 tablet_replica.cc:330] T edc8fa36097c49c88c1b5b981a32e2a2 P 1cb67158133e4648a6eceba32d4dbba3: stopping tablet replica
I20240502 14:14:52.526541 30017 raft_consensus.cc:2229] T edc8fa36097c49c88c1b5b981a32e2a2 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:52.526578 30075 raft_consensus.cc:2229] T edc8fa36097c49c88c1b5b981a32e2a2 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:52.526775 30075 raft_consensus.cc:2258] T edc8fa36097c49c88c1b5b981a32e2a2 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.526784 30017 raft_consensus.cc:2258] T edc8fa36097c49c88c1b5b981a32e2a2 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.527801 30075 ts_tablet_manager.cc:1875] T edc8fa36097c49c88c1b5b981a32e2a2 P 1cb67158133e4648a6eceba32d4dbba3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.527823 30019 ts_tablet_manager.cc:1875] T edc8fa36097c49c88c1b5b981a32e2a2 P e4cfb8fc5a474177af6848a6efa13a27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.528235 30017 ts_tablet_manager.cc:1875] T edc8fa36097c49c88c1b5b981a32e2a2 P 89af7f423cf44717a08d6b142f0ed49d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.534848 29943 consensus_queue.cc:1035] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:52.535240 30019 ts_tablet_manager.cc:1888] T edc8fa36097c49c88c1b5b981a32e2a2 P e4cfb8fc5a474177af6848a6efa13a27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:52.535454 30019 log.cc:1196] T edc8fa36097c49c88c1b5b981a32e2a2 P e4cfb8fc5a474177af6848a6efa13a27: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/wals/edc8fa36097c49c88c1b5b981a32e2a2
I20240502 14:14:52.536089 30019 ts_tablet_manager.cc:1909] T edc8fa36097c49c88c1b5b981a32e2a2 P e4cfb8fc5a474177af6848a6efa13a27: Deleting consensus metadata
I20240502 14:14:52.539326 30017 ts_tablet_manager.cc:1888] T edc8fa36097c49c88c1b5b981a32e2a2 P 89af7f423cf44717a08d6b142f0ed49d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:52.539644 30017 log.cc:1196] T edc8fa36097c49c88c1b5b981a32e2a2 P 89af7f423cf44717a08d6b142f0ed49d: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/edc8fa36097c49c88c1b5b981a32e2a2
I20240502 14:14:52.540378 30017 ts_tablet_manager.cc:1909] T edc8fa36097c49c88c1b5b981a32e2a2 P 89af7f423cf44717a08d6b142f0ed49d: Deleting consensus metadata
I20240502 14:14:52.543476 26271 catalog_manager.cc:4830] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): tablet edc8fa36097c49c88c1b5b981a32e2a2 (table rebalancer_dynamic_table_new_name [id=02258aeb7420405ab4cce9f74610170f]) successfully deleted
I20240502 14:14:52.544463 26272 catalog_manager.cc:4830] TS e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): tablet edc8fa36097c49c88c1b5b981a32e2a2 (table rebalancer_dynamic_table_new_name [id=02258aeb7420405ab4cce9f74610170f]) successfully deleted
I20240502 14:14:52.553694 29791 consensus_queue.cc:1035] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:52.556815 30075 ts_tablet_manager.cc:1888] T edc8fa36097c49c88c1b5b981a32e2a2 P 1cb67158133e4648a6eceba32d4dbba3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:52.557145 30075 log.cc:1196] T edc8fa36097c49c88c1b5b981a32e2a2 P 1cb67158133e4648a6eceba32d4dbba3: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal/wals/edc8fa36097c49c88c1b5b981a32e2a2
I20240502 14:14:52.559361 30075 ts_tablet_manager.cc:1909] T edc8fa36097c49c88c1b5b981a32e2a2 P 1cb67158133e4648a6eceba32d4dbba3: Deleting consensus metadata
I20240502 14:14:52.563024 29694 consensus_queue.cc:1035] T 2532ebd7bdf14876b74a26afd674a7ac P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:52.566164 26271 catalog_manager.cc:4830] TS 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): tablet edc8fa36097c49c88c1b5b981a32e2a2 (table rebalancer_dynamic_table_new_name [id=02258aeb7420405ab4cce9f74610170f]) successfully deleted
I20240502 14:14:52.570439 26427 consensus_queue.cc:237] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:52.595419 26818 raft_consensus.cc:1261] T bb3e4ec75888419aa4e065cf5513ae5b P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:14:52.596068 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bb3e4ec75888419aa4e065cf5513ae5b. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:52.596832 30041 consensus_queue.cc:1035] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:52.604812 29694 consensus_queue.cc:1035] T 2532ebd7bdf14876b74a26afd674a7ac P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:52.607046 26558 raft_consensus.cc:1261] T bb3e4ec75888419aa4e065cf5513ae5b P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:52.608146 30041 consensus_queue.cc:1035] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:52.612529 29652 consensus_queue.cc:1035] T 01e6bc3689db46e2ad7d0484a29af287 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:52.619889 30039 raft_consensus.cc:2932] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:14:52.626050 26819 raft_consensus.cc:2932] T bb3e4ec75888419aa4e065cf5513ae5b P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:14:52.629689 26558 raft_consensus.cc:2932] T bb3e4ec75888419aa4e065cf5513ae5b P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:14:52.630673 26279 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:14:52.636780 30008 consensus_queue.cc:1035] T 01e6bc3689db46e2ad7d0484a29af287 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:52.636874 26283 catalog_manager.cc:5484] T bb3e4ec75888419aa4e065cf5513ae5b P ba187e38a2754d60a12de9ac4856430a reported cstate change: config changed from index -1 to 4, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:14:52.657119 29155 tablet_service.cc:1467] Processing CreateTablet for tablet 1a3e938ab64e4fc09462c2dbfcee015e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=58dd440e1ee2463cb0df003412402dcd]), partition=
I20240502 14:14:52.657150 29554 tablet_service.cc:1467] Processing CreateTablet for tablet 1a3e938ab64e4fc09462c2dbfcee015e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=58dd440e1ee2463cb0df003412402dcd]), partition=
I20240502 14:14:52.657572 29554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a3e938ab64e4fc09462c2dbfcee015e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:52.657596 29155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a3e938ab64e4fc09462c2dbfcee015e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:52.663880 29288 tablet_service.cc:1467] Processing CreateTablet for tablet 1a3e938ab64e4fc09462c2dbfcee015e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=58dd440e1ee2463cb0df003412402dcd]), partition=
I20240502 14:14:52.664335 29288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a3e938ab64e4fc09462c2dbfcee015e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:52.682673 30097 tablet_bootstrap.cc:492] T 1a3e938ab64e4fc09462c2dbfcee015e P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:14:52.696624 30097 tablet_bootstrap.cc:654] T 1a3e938ab64e4fc09462c2dbfcee015e P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:52.708297 30097 tablet_bootstrap.cc:492] T 1a3e938ab64e4fc09462c2dbfcee015e P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:14:52.709146 30097 ts_tablet_manager.cc:1367] T 1a3e938ab64e4fc09462c2dbfcee015e P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.000s
I20240502 14:14:52.710366 30097 raft_consensus.cc:348] T 1a3e938ab64e4fc09462c2dbfcee015e P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:52.710772 30097 raft_consensus.cc:374] T 1a3e938ab64e4fc09462c2dbfcee015e P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:52.710935 30097 raft_consensus.cc:729] T 1a3e938ab64e4fc09462c2dbfcee015e P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:14:52.711372 30097 consensus_queue.cc:260] T 1a3e938ab64e4fc09462c2dbfcee015e P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:52.712173 30097 ts_tablet_manager.cc:1398] T 1a3e938ab64e4fc09462c2dbfcee015e P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:52.720227 30102 tablet_bootstrap.cc:492] T 1a3e938ab64e4fc09462c2dbfcee015e P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:14:52.723345 30101 tablet_bootstrap.cc:492] T 1a3e938ab64e4fc09462c2dbfcee015e P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:14:52.727335 30102 tablet_bootstrap.cc:654] T 1a3e938ab64e4fc09462c2dbfcee015e P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:52.729890 30101 tablet_bootstrap.cc:654] T 1a3e938ab64e4fc09462c2dbfcee015e P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:52.736963 30102 tablet_bootstrap.cc:492] T 1a3e938ab64e4fc09462c2dbfcee015e P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:14:52.737308 30102 ts_tablet_manager.cc:1367] T 1a3e938ab64e4fc09462c2dbfcee015e P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.000s
I20240502 14:14:52.738343 30102 raft_consensus.cc:348] T 1a3e938ab64e4fc09462c2dbfcee015e P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:52.739925 30102 raft_consensus.cc:374] T 1a3e938ab64e4fc09462c2dbfcee015e P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:52.740101 30102 raft_consensus.cc:729] T 1a3e938ab64e4fc09462c2dbfcee015e P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:14:52.740569 30102 consensus_queue.cc:260] T 1a3e938ab64e4fc09462c2dbfcee015e P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:52.745030 30101 tablet_bootstrap.cc:492] T 1a3e938ab64e4fc09462c2dbfcee015e P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:14:52.745227 30101 ts_tablet_manager.cc:1367] T 1a3e938ab64e4fc09462c2dbfcee015e P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.000s
I20240502 14:14:52.746367 30101 raft_consensus.cc:348] T 1a3e938ab64e4fc09462c2dbfcee015e P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:52.746675 30101 raft_consensus.cc:374] T 1a3e938ab64e4fc09462c2dbfcee015e P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:52.746802 30101 raft_consensus.cc:729] T 1a3e938ab64e4fc09462c2dbfcee015e P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:14:52.747102 30101 consensus_queue.cc:260] T 1a3e938ab64e4fc09462c2dbfcee015e P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:52.749171 30102 ts_tablet_manager.cc:1398] T 1a3e938ab64e4fc09462c2dbfcee015e P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20240502 14:14:52.750941 30101 ts_tablet_manager.cc:1398] T 1a3e938ab64e4fc09462c2dbfcee015e P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240502 14:14:52.780172 26280 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_extra_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:52.796830 29155 tablet_service.cc:1467] Processing CreateTablet for tablet 83dd34749ff8496483365b30a86db71f (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=7ae78c7dfcaf4548997196f534416ea8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:52.797380 29155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83dd34749ff8496483365b30a86db71f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:52.797883 26538 tablet_service.cc:1467] Processing CreateTablet for tablet 253449163c65471d8736b92b25da0f92 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=7ae78c7dfcaf4548997196f534416ea8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:52.798332 26538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 253449163c65471d8736b92b25da0f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:52.799582 29554 tablet_service.cc:1467] Processing CreateTablet for tablet 83dd34749ff8496483365b30a86db71f (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=7ae78c7dfcaf4548997196f534416ea8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:52.800040 29554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83dd34749ff8496483365b30a86db71f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:52.800876 29287 tablet_service.cc:1467] Processing CreateTablet for tablet 56285a2fdcdd47e9b9c81b4b8993c4aa (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=7ae78c7dfcaf4548997196f534416ea8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:52.801290 29287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56285a2fdcdd47e9b9c81b4b8993c4aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:52.801331 29156 tablet_service.cc:1467] Processing CreateTablet for tablet 253449163c65471d8736b92b25da0f92 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=7ae78c7dfcaf4548997196f534416ea8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:52.801745 29156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 253449163c65471d8736b92b25da0f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:52.804885 29422 tablet_service.cc:1467] Processing CreateTablet for tablet 83dd34749ff8496483365b30a86db71f (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=7ae78c7dfcaf4548997196f534416ea8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:52.805147 29288 tablet_service.cc:1467] Processing CreateTablet for tablet 253449163c65471d8736b92b25da0f92 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=7ae78c7dfcaf4548997196f534416ea8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:52.805140 26406 tablet_service.cc:1467] Processing CreateTablet for tablet 56285a2fdcdd47e9b9c81b4b8993c4aa (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=7ae78c7dfcaf4548997196f534416ea8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:52.805392 29422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83dd34749ff8496483365b30a86db71f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:52.805605 26406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56285a2fdcdd47e9b9c81b4b8993c4aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:52.806759 29288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 253449163c65471d8736b92b25da0f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:52.808954 26536 tablet_service.cc:1467] Processing CreateTablet for tablet 56285a2fdcdd47e9b9c81b4b8993c4aa (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=7ae78c7dfcaf4548997196f534416ea8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:52.809379 26536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56285a2fdcdd47e9b9c81b4b8993c4aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:52.821805 30102 tablet_bootstrap.cc:492] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:14:52.822630 30097 tablet_bootstrap.cc:492] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:14:52.822873 30111 tablet_bootstrap.cc:492] T 253449163c65471d8736b92b25da0f92 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:52.823710 30102 tablet_bootstrap.cc:654] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:52.824362 30097 tablet_bootstrap.cc:654] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:52.824442 30101 tablet_bootstrap.cc:492] T 253449163c65471d8736b92b25da0f92 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:14:52.826066 30101 tablet_bootstrap.cc:654] T 253449163c65471d8736b92b25da0f92 P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:52.828341 30111 tablet_bootstrap.cc:654] T 253449163c65471d8736b92b25da0f92 P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:52.829249 30101 tablet_bootstrap.cc:492] T 253449163c65471d8736b92b25da0f92 P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:14:52.829439 30101 ts_tablet_manager.cc:1367] T 253449163c65471d8736b92b25da0f92 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 14:14:52.830078 30101 raft_consensus.cc:348] T 253449163c65471d8736b92b25da0f92 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:52.830292 30101 raft_consensus.cc:374] T 253449163c65471d8736b92b25da0f92 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:52.830374 30101 raft_consensus.cc:729] T 253449163c65471d8736b92b25da0f92 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:14:52.830658 30101 consensus_queue.cc:260] T 253449163c65471d8736b92b25da0f92 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:52.831416 30101 ts_tablet_manager.cc:1398] T 253449163c65471d8736b92b25da0f92 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:52.831811 30101 tablet_bootstrap.cc:492] T 56285a2fdcdd47e9b9c81b4b8993c4aa P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:14:52.840556 30097 tablet_bootstrap.cc:492] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:14:52.840796 30097 ts_tablet_manager.cc:1367] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.006s
I20240502 14:14:52.841897 30097 raft_consensus.cc:348] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:52.842262 30097 raft_consensus.cc:374] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:52.842366 30097 raft_consensus.cc:729] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:14:52.842746 30101 tablet_bootstrap.cc:654] T 56285a2fdcdd47e9b9c81b4b8993c4aa P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:52.842705 30097 consensus_queue.cc:260] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:52.843509 30097 ts_tablet_manager.cc:1398] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20240502 14:14:52.844734 30112 tablet_bootstrap.cc:492] T 83dd34749ff8496483365b30a86db71f P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:14:52.849637 30102 tablet_bootstrap.cc:492] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:14:52.849964 30102 ts_tablet_manager.cc:1367] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.010s
I20240502 14:14:52.851074 30102 raft_consensus.cc:348] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:52.851430 30102 raft_consensus.cc:374] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:52.851601 30102 raft_consensus.cc:729] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:14:52.851994 30102 consensus_queue.cc:260] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:52.854336 30118 ts_tablet_manager.cc:902] T 0fc2e6c74dc34a45ae1ec46608642880 P 1cb67158133e4648a6eceba32d4dbba3: Initiating tablet copy from peer 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:14:52.856303 30102 ts_tablet_manager.cc:1398] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:14:52.860615 30102 tablet_bootstrap.cc:492] T 83dd34749ff8496483365b30a86db71f P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:14:52.862031 30112 tablet_bootstrap.cc:654] T 83dd34749ff8496483365b30a86db71f P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:52.862321 30113 tablet_bootstrap.cc:492] T 56285a2fdcdd47e9b9c81b4b8993c4aa P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:52.863193 30111 tablet_bootstrap.cc:492] T 253449163c65471d8736b92b25da0f92 P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:52.863445 30111 ts_tablet_manager.cc:1367] T 253449163c65471d8736b92b25da0f92 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.041s user 0.006s sys 0.006s
I20240502 14:14:52.863852 30101 tablet_bootstrap.cc:492] T 56285a2fdcdd47e9b9c81b4b8993c4aa P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:14:52.864054 30101 ts_tablet_manager.cc:1367] T 56285a2fdcdd47e9b9c81b4b8993c4aa P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.008s
I20240502 14:14:52.864555 30111 raft_consensus.cc:348] T 253449163c65471d8736b92b25da0f92 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:52.864856 30111 raft_consensus.cc:374] T 253449163c65471d8736b92b25da0f92 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:52.865371 30111 raft_consensus.cc:729] T 253449163c65471d8736b92b25da0f92 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:52.865705 30111 consensus_queue.cc:260] T 253449163c65471d8736b92b25da0f92 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:52.866665 30111 ts_tablet_manager.cc:1398] T 253449163c65471d8736b92b25da0f92 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:52.866586 30101 raft_consensus.cc:348] T 56285a2fdcdd47e9b9c81b4b8993c4aa P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:52.866969 30101 raft_consensus.cc:374] T 56285a2fdcdd47e9b9c81b4b8993c4aa P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:52.867121 30101 raft_consensus.cc:729] T 56285a2fdcdd47e9b9c81b4b8993c4aa P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:14:52.867301 30111 tablet_bootstrap.cc:492] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:52.867447 30101 consensus_queue.cc:260] T 56285a2fdcdd47e9b9c81b4b8993c4aa P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:52.869405 30111 tablet_bootstrap.cc:654] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:52.870126 30101 ts_tablet_manager.cc:1398] T 56285a2fdcdd47e9b9c81b4b8993c4aa P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 14:14:52.871693 30102 tablet_bootstrap.cc:654] T 83dd34749ff8496483365b30a86db71f P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:52.873412 29872 raft_consensus.cc:482] T 1a3e938ab64e4fc09462c2dbfcee015e P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:52.873658 29872 raft_consensus.cc:504] T 1a3e938ab64e4fc09462c2dbfcee015e P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:52.874528 29872 leader_election.cc:288] T 1a3e938ab64e4fc09462c2dbfcee015e P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:14:52.875181 30111 tablet_bootstrap.cc:492] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:52.875463 29174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a3e938ab64e4fc09462c2dbfcee015e" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:14:52.875788 29174 raft_consensus.cc:2454] T 1a3e938ab64e4fc09462c2dbfcee015e P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 0.
I20240502 14:14:52.876291 29574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a3e938ab64e4fc09462c2dbfcee015e" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86" is_pre_election: true
I20240502 14:14:52.876576 29574 raft_consensus.cc:2454] T 1a3e938ab64e4fc09462c2dbfcee015e P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 0.
I20240502 14:14:52.876760 30111 ts_tablet_manager.cc:1367] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20240502 14:14:52.877785 29244 leader_election.cc:302] T 1a3e938ab64e4fc09462c2dbfcee015e P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:14:52.881187 30111 raft_consensus.cc:348] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:52.881635 30111 raft_consensus.cc:374] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:52.882181 30111 raft_consensus.cc:729] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:52.882514 30111 consensus_queue.cc:260] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:52.883131 29872 raft_consensus.cc:2781] T 1a3e938ab64e4fc09462c2dbfcee015e P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:52.883255 29872 raft_consensus.cc:482] T 1a3e938ab64e4fc09462c2dbfcee015e P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:52.883277 30113 tablet_bootstrap.cc:654] T 56285a2fdcdd47e9b9c81b4b8993c4aa P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:52.883258 30111 ts_tablet_manager.cc:1398] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 14:14:52.883366 29872 raft_consensus.cc:3037] T 1a3e938ab64e4fc09462c2dbfcee015e P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:52.886643 29872 raft_consensus.cc:504] T 1a3e938ab64e4fc09462c2dbfcee015e P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:52.887444 30118 tablet_copy_client.cc:308] T 0fc2e6c74dc34a45ae1ec46608642880 P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.193:37985
I20240502 14:14:52.888760 26447 tablet_copy_service.cc:140] P 89af7f423cf44717a08d6b142f0ed49d: Received BeginTabletCopySession request for tablet 0fc2e6c74dc34a45ae1ec46608642880 from peer 1cb67158133e4648a6eceba32d4dbba3 ({username='slave'} at 127.19.94.197:56601)
I20240502 14:14:52.889073 26447 tablet_copy_service.cc:161] P 89af7f423cf44717a08d6b142f0ed49d: Beginning new tablet copy session on tablet 0fc2e6c74dc34a45ae1ec46608642880 from peer 1cb67158133e4648a6eceba32d4dbba3 at {username='slave'} at 127.19.94.197:56601: session id = 1cb67158133e4648a6eceba32d4dbba3-0fc2e6c74dc34a45ae1ec46608642880
I20240502 14:14:52.893074 29574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a3e938ab64e4fc09462c2dbfcee015e" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86"
I20240502 14:14:52.893335 29574 raft_consensus.cc:3037] T 1a3e938ab64e4fc09462c2dbfcee015e P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:52.893585 30113 tablet_bootstrap.cc:492] T 56285a2fdcdd47e9b9c81b4b8993c4aa P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:52.893749 30113 ts_tablet_manager.cc:1367] T 56285a2fdcdd47e9b9c81b4b8993c4aa P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.002s
I20240502 14:14:52.894627 30113 raft_consensus.cc:348] T 56285a2fdcdd47e9b9c81b4b8993c4aa P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:52.894846 30113 raft_consensus.cc:374] T 56285a2fdcdd47e9b9c81b4b8993c4aa P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:52.894930 30113 raft_consensus.cc:729] T 56285a2fdcdd47e9b9c81b4b8993c4aa P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:52.895169 30113 consensus_queue.cc:260] T 56285a2fdcdd47e9b9c81b4b8993c4aa P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:52.896353 30113 ts_tablet_manager.cc:1398] T 56285a2fdcdd47e9b9c81b4b8993c4aa P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:14:52.900779 29574 raft_consensus.cc:2454] T 1a3e938ab64e4fc09462c2dbfcee015e P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 1.
I20240502 14:14:52.908728 26447 tablet_copy_source_session.cc:205] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:52.909116 30112 tablet_bootstrap.cc:492] T 83dd34749ff8496483365b30a86db71f P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:14:52.909279 30112 ts_tablet_manager.cc:1367] T 83dd34749ff8496483365b30a86db71f P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.065s user 0.002s sys 0.002s
I20240502 14:14:52.887594 29872 leader_election.cc:288] T 1a3e938ab64e4fc09462c2dbfcee015e P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 election: Requested vote from peers 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:14:52.902096 29245 leader_election.cc:302] T 1a3e938ab64e4fc09462c2dbfcee015e P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:14:52.909848 29872 raft_consensus.cc:482] T 253449163c65471d8736b92b25da0f92 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:52.910005 29872 raft_consensus.cc:504] T 253449163c65471d8736b92b25da0f92 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:52.910290 30112 raft_consensus.cc:348] T 83dd34749ff8496483365b30a86db71f P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:52.910509 30112 raft_consensus.cc:374] T 83dd34749ff8496483365b30a86db71f P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:52.910601 30112 raft_consensus.cc:729] T 83dd34749ff8496483365b30a86db71f P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:14:52.910833 30112 consensus_queue.cc:260] T 83dd34749ff8496483365b30a86db71f P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:52.911468 30112 ts_tablet_manager.cc:1398] T 83dd34749ff8496483365b30a86db71f P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:14:52.914625 29872 leader_election.cc:288] T 253449163c65471d8736b92b25da0f92 P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:14:52.915621 30118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fc2e6c74dc34a45ae1ec46608642880. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:52.915927 29872 raft_consensus.cc:2781] T 1a3e938ab64e4fc09462c2dbfcee015e P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:52.916992 29872 raft_consensus.cc:686] T 1a3e938ab64e4fc09462c2dbfcee015e P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 LEADER]: Becoming Leader. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Running, Role: LEADER
I20240502 14:14:52.918071 26283 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240502 14:14:52.918318 26283 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240502 14:14:52.918823 29872 consensus_queue.cc:237] T 1a3e938ab64e4fc09462c2dbfcee015e P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:52.923257 26283 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet 82737d75c8a64efe98cab1e21160d9b6
I20240502 14:14:52.924822 26797 tablet_service.cc:1514] Processing DeleteTablet for tablet 82737d75c8a64efe98cab1e21160d9b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:52 UTC) from {username='slave'} at 127.0.0.1:56010
I20240502 14:14:52.925153 30019 tablet_replica.cc:330] T 82737d75c8a64efe98cab1e21160d9b6 P e4cfb8fc5a474177af6848a6efa13a27: stopping tablet replica
I20240502 14:14:52.925376 30019 raft_consensus.cc:2229] T 82737d75c8a64efe98cab1e21160d9b6 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:52.925536 29918 raft_consensus.cc:482] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:52.925766 30019 raft_consensus.cc:2258] T 82737d75c8a64efe98cab1e21160d9b6 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.925745 29918 raft_consensus.cc:504] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:52.926398 30019 ts_tablet_manager.cc:1875] T 82737d75c8a64efe98cab1e21160d9b6 P e4cfb8fc5a474177af6848a6efa13a27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.926671 29918 leader_election.cc:288] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:14:52.927332 26557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253449163c65471d8736b92b25da0f92" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:52.927598 26557 raft_consensus.cc:2454] T 253449163c65471d8736b92b25da0f92 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 0.
I20240502 14:14:52.928761 29111 leader_election.cc:302] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:52.928895 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253449163c65471d8736b92b25da0f92" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:14:52.929093 29918 raft_consensus.cc:2781] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:52.929128 29309 raft_consensus.cc:2454] T 253449163c65471d8736b92b25da0f92 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 0.
I20240502 14:14:52.929193 29918 raft_consensus.cc:482] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:52.929299 29918 raft_consensus.cc:3037] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:52.931342 26536 tablet_service.cc:1514] Processing DeleteTablet for tablet 82737d75c8a64efe98cab1e21160d9b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:52 UTC) from {username='slave'} at 127.0.0.1:51696
I20240502 14:14:52.931672 30018 tablet_replica.cc:330] T 82737d75c8a64efe98cab1e21160d9b6 P ba187e38a2754d60a12de9ac4856430a: stopping tablet replica
I20240502 14:14:52.931890 30018 raft_consensus.cc:2229] T 82737d75c8a64efe98cab1e21160d9b6 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:52.932184 30018 raft_consensus.cc:2258] T 82737d75c8a64efe98cab1e21160d9b6 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.932404 26283 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet 2b50fc58a5384c6b8d2c78b88e0b3d95
I20240502 14:14:52.932310 26557 consensus_queue.cc:237] T 825ae3e90e214a6f98394765edf6ff2d P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:52.933074 30018 ts_tablet_manager.cc:1875] T 82737d75c8a64efe98cab1e21160d9b6 P ba187e38a2754d60a12de9ac4856430a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.933158 26667 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b50fc58a5384c6b8d2c78b88e0b3d95 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:52 UTC) from {username='slave'} at 127.0.0.1:34150
I20240502 14:14:52.933454 30020 tablet_replica.cc:330] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: stopping tablet replica
I20240502 14:14:52.933683 30020 raft_consensus.cc:2229] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:52.934013 30020 raft_consensus.cc:2258] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.934767 30020 ts_tablet_manager.cc:1875] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.935881 26407 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b50fc58a5384c6b8d2c78b88e0b3d95 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:52 UTC) from {username='slave'} at 127.0.0.1:37422
I20240502 14:14:52.936184 30017 tablet_replica.cc:330] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 89af7f423cf44717a08d6b142f0ed49d: stopping tablet replica
I20240502 14:14:52.936435 30017 raft_consensus.cc:2229] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:52.936493 26283 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet 5d8ed238257d46b9b78f0f8a19f2adba
I20240502 14:14:52.936614 26536 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b50fc58a5384c6b8d2c78b88e0b3d95 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:52 UTC) from {username='slave'} at 127.0.0.1:51696
I20240502 14:14:52.936674 30017 raft_consensus.cc:2258] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.937455 30017 ts_tablet_manager.cc:1875] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 89af7f423cf44717a08d6b142f0ed49d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.937441 26797 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d8ed238257d46b9b78f0f8a19f2adba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:52 UTC) from {username='slave'} at 127.0.0.1:56010
I20240502 14:14:52.939203 26667 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d8ed238257d46b9b78f0f8a19f2adba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:52 UTC) from {username='slave'} at 127.0.0.1:34150
I20240502 14:14:52.940038 26536 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d8ed238257d46b9b78f0f8a19f2adba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:52 UTC) from {username='slave'} at 127.0.0.1:51696
I20240502 14:14:52.941704 26406 tablet_service.cc:1514] Processing DeleteTablet for tablet 82737d75c8a64efe98cab1e21160d9b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:52 UTC) from {username='slave'} at 127.0.0.1:37422
I20240502 14:14:52.942991 29918 raft_consensus.cc:504] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:52.943778 29918 leader_election.cc:288] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 election: Requested vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:14:52.944314 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253449163c65471d8736b92b25da0f92" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:14:52.944521 30020 ts_tablet_manager.cc:1888] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:52.944603 26558 raft_consensus.cc:3037] T 253449163c65471d8736b92b25da0f92 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:52.944572 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253449163c65471d8736b92b25da0f92" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
I20240502 14:14:52.944684 30020 log.cc:1196] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/wals/2b50fc58a5384c6b8d2c78b88e0b3d95
I20240502 14:14:52.944821 29309 raft_consensus.cc:3037] T 253449163c65471d8736b92b25da0f92 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:52.947531 26558 raft_consensus.cc:2454] T 253449163c65471d8736b92b25da0f92 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 1.
I20240502 14:14:52.947690 29309 raft_consensus.cc:2454] T 253449163c65471d8736b92b25da0f92 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 1.
I20240502 14:14:52.947842 30020 ts_tablet_manager.cc:1909] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting consensus metadata
W20240502 14:14:52.948794 26604 tablet.cc:2369] T 253449163c65471d8736b92b25da0f92 P ba187e38a2754d60a12de9ac4856430a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:14:52.949363 26272 catalog_manager.cc:4830] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): tablet 2b50fc58a5384c6b8d2c78b88e0b3d95 (table rebalance_test_table_16 [id=f1e711a462c54504b4fe5c866615f824]) successfully deleted
I20240502 14:14:52.949790 30020 tablet_replica.cc:330] T 5d8ed238257d46b9b78f0f8a19f2adba P 69ab78b2a1b74d2bbf2cdeafa0a4af23: stopping tablet replica
I20240502 14:14:52.950008 30020 raft_consensus.cc:2229] T 5d8ed238257d46b9b78f0f8a19f2adba P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:52.950346 30020 raft_consensus.cc:2258] T 5d8ed238257d46b9b78f0f8a19f2adba P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.950769 30118 tablet_copy_client.cc:783] T 0fc2e6c74dc34a45ae1ec46608642880 P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:52.951100 30020 ts_tablet_manager.cc:1875] T 5d8ed238257d46b9b78f0f8a19f2adba P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.951164 30118 tablet_copy_client.cc:647] T 0fc2e6c74dc34a45ae1ec46608642880 P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:52.951627 30019 ts_tablet_manager.cc:1888] T 82737d75c8a64efe98cab1e21160d9b6 P e4cfb8fc5a474177af6848a6efa13a27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:52.951855 30019 log.cc:1196] T 82737d75c8a64efe98cab1e21160d9b6 P e4cfb8fc5a474177af6848a6efa13a27: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/wals/82737d75c8a64efe98cab1e21160d9b6
I20240502 14:14:52.952109 30018 ts_tablet_manager.cc:1888] T 82737d75c8a64efe98cab1e21160d9b6 P ba187e38a2754d60a12de9ac4856430a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:52.952454 30019 ts_tablet_manager.cc:1909] T 82737d75c8a64efe98cab1e21160d9b6 P e4cfb8fc5a474177af6848a6efa13a27: Deleting consensus metadata
I20240502 14:14:52.952909 30017 ts_tablet_manager.cc:1888] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 89af7f423cf44717a08d6b142f0ed49d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:52.953544 30017 log.cc:1196] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 89af7f423cf44717a08d6b142f0ed49d: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/2b50fc58a5384c6b8d2c78b88e0b3d95
W20240502 14:14:52.953974 26473 tablet.cc:2369] T 56285a2fdcdd47e9b9c81b4b8993c4aa P 89af7f423cf44717a08d6b142f0ed49d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:14:52.954087 26272 catalog_manager.cc:4830] TS e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): tablet 82737d75c8a64efe98cab1e21160d9b6 (table rebalance_test_table_16 [id=f1e711a462c54504b4fe5c866615f824]) successfully deleted
I20240502 14:14:52.954178 30019 tablet_replica.cc:330] T 5d8ed238257d46b9b78f0f8a19f2adba P e4cfb8fc5a474177af6848a6efa13a27: stopping tablet replica
I20240502 14:14:52.954454 30019 raft_consensus.cc:2229] T 5d8ed238257d46b9b78f0f8a19f2adba P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:52.954721 30019 raft_consensus.cc:2258] T 5d8ed238257d46b9b78f0f8a19f2adba P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.954205 30017 ts_tablet_manager.cc:1909] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P 89af7f423cf44717a08d6b142f0ed49d: Deleting consensus metadata
I20240502 14:14:52.955566 30019 ts_tablet_manager.cc:1875] T 5d8ed238257d46b9b78f0f8a19f2adba P e4cfb8fc5a474177af6848a6efa13a27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.956185 30018 log.cc:1196] T 82737d75c8a64efe98cab1e21160d9b6 P ba187e38a2754d60a12de9ac4856430a: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/82737d75c8a64efe98cab1e21160d9b6
I20240502 14:14:52.956537 30017 tablet_replica.cc:330] T 82737d75c8a64efe98cab1e21160d9b6 P 89af7f423cf44717a08d6b142f0ed49d: stopping tablet replica
I20240502 14:14:52.957453 26271 catalog_manager.cc:4830] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): tablet 2b50fc58a5384c6b8d2c78b88e0b3d95 (table rebalance_test_table_16 [id=f1e711a462c54504b4fe5c866615f824]) successfully deleted
I20240502 14:14:52.957511 30017 raft_consensus.cc:2229] T 82737d75c8a64efe98cab1e21160d9b6 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:52.958668 30017 raft_consensus.cc:2258] T 82737d75c8a64efe98cab1e21160d9b6 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.959940 30018 ts_tablet_manager.cc:1909] T 82737d75c8a64efe98cab1e21160d9b6 P ba187e38a2754d60a12de9ac4856430a: Deleting consensus metadata
I20240502 14:14:52.962131 26271 catalog_manager.cc:4830] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): tablet 82737d75c8a64efe98cab1e21160d9b6 (table rebalance_test_table_16 [id=f1e711a462c54504b4fe5c866615f824]) successfully deleted
I20240502 14:14:52.963466 30017 ts_tablet_manager.cc:1875] T 82737d75c8a64efe98cab1e21160d9b6 P 89af7f423cf44717a08d6b142f0ed49d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.965013 30018 tablet_replica.cc:330] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P ba187e38a2754d60a12de9ac4856430a: stopping tablet replica
I20240502 14:14:52.965384 30018 raft_consensus.cc:2229] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:52.966121 30019 ts_tablet_manager.cc:1888] T 5d8ed238257d46b9b78f0f8a19f2adba P e4cfb8fc5a474177af6848a6efa13a27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:52.966408 30019 log.cc:1196] T 5d8ed238257d46b9b78f0f8a19f2adba P e4cfb8fc5a474177af6848a6efa13a27: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/wals/5d8ed238257d46b9b78f0f8a19f2adba
I20240502 14:14:52.967718 30019 ts_tablet_manager.cc:1909] T 5d8ed238257d46b9b78f0f8a19f2adba P e4cfb8fc5a474177af6848a6efa13a27: Deleting consensus metadata
I20240502 14:14:52.968019 30020 ts_tablet_manager.cc:1888] T 5d8ed238257d46b9b78f0f8a19f2adba P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:52.969022 26272 catalog_manager.cc:4830] TS e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): tablet 5d8ed238257d46b9b78f0f8a19f2adba (table rebalance_test_table_16 [id=f1e711a462c54504b4fe5c866615f824]) successfully deleted
I20240502 14:14:52.969839 30018 raft_consensus.cc:2258] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.970854 30018 ts_tablet_manager.cc:1875] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P ba187e38a2754d60a12de9ac4856430a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.972533 30017 ts_tablet_manager.cc:1888] T 82737d75c8a64efe98cab1e21160d9b6 P 89af7f423cf44717a08d6b142f0ed49d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:52.972793 30017 log.cc:1196] T 82737d75c8a64efe98cab1e21160d9b6 P 89af7f423cf44717a08d6b142f0ed49d: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/82737d75c8a64efe98cab1e21160d9b6
I20240502 14:14:52.972878 30020 log.cc:1196] T 5d8ed238257d46b9b78f0f8a19f2adba P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/wals/5d8ed238257d46b9b78f0f8a19f2adba
I20240502 14:14:52.973496 30017 ts_tablet_manager.cc:1909] T 82737d75c8a64efe98cab1e21160d9b6 P 89af7f423cf44717a08d6b142f0ed49d: Deleting consensus metadata
I20240502 14:14:52.973490 30020 ts_tablet_manager.cc:1909] T 5d8ed238257d46b9b78f0f8a19f2adba P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting consensus metadata
I20240502 14:14:52.974997 26271 catalog_manager.cc:4830] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): tablet 82737d75c8a64efe98cab1e21160d9b6 (table rebalance_test_table_16 [id=f1e711a462c54504b4fe5c866615f824]) successfully deleted
I20240502 14:14:52.973738 29111 leader_election.cc:302] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:52.975387 29918 raft_consensus.cc:2781] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:52.975579 29918 raft_consensus.cc:686] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 LEADER]: Becoming Leader. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Running, Role: LEADER
I20240502 14:14:52.975888 29918 consensus_queue.cc:237] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:52.976786 26272 catalog_manager.cc:4830] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): tablet 5d8ed238257d46b9b78f0f8a19f2adba (table rebalance_test_table_16 [id=f1e711a462c54504b4fe5c866615f824]) successfully deleted
I20240502 14:14:52.978060 30118 tablet_copy_client.cc:523] T 0fc2e6c74dc34a45ae1ec46608642880 P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:52.983386 30118 tablet_bootstrap.cc:492] T 0fc2e6c74dc34a45ae1ec46608642880 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:14:52.984164 29174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a3e938ab64e4fc09462c2dbfcee015e" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:14:52.984539 29174 raft_consensus.cc:3037] T 1a3e938ab64e4fc09462c2dbfcee015e P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:52.984674 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253449163c65471d8736b92b25da0f92" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:52.984980 26558 raft_consensus.cc:2379] T 253449163c65471d8736b92b25da0f92 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c1100e16b3b9495ca61d4f95e89b6a89 in current term 1: Already voted for candidate 1cb67158133e4648a6eceba32d4dbba3 in this term.
I20240502 14:14:52.987011 26280 catalog_manager.cc:5484] T 1a3e938ab64e4fc09462c2dbfcee015e P c1100e16b3b9495ca61d4f95e89b6a89 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198). New cstate: current_term: 1 leader_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:14:52.989876 29176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253449163c65471d8736b92b25da0f92" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:14:52.992378 26283 catalog_manager.cc:5484] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197). New cstate: current_term: 1 leader_uuid: "1cb67158133e4648a6eceba32d4dbba3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:14:52.993574 29174 raft_consensus.cc:2454] T 1a3e938ab64e4fc09462c2dbfcee015e P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 1.
I20240502 14:14:52.994696 30018 ts_tablet_manager.cc:1888] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P ba187e38a2754d60a12de9ac4856430a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:52.994939 30018 log.cc:1196] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P ba187e38a2754d60a12de9ac4856430a: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/2b50fc58a5384c6b8d2c78b88e0b3d95
I20240502 14:14:52.995523 30018 ts_tablet_manager.cc:1909] T 2b50fc58a5384c6b8d2c78b88e0b3d95 P ba187e38a2754d60a12de9ac4856430a: Deleting consensus metadata
I20240502 14:14:52.996634 30018 tablet_replica.cc:330] T 5d8ed238257d46b9b78f0f8a19f2adba P ba187e38a2754d60a12de9ac4856430a: stopping tablet replica
I20240502 14:14:52.996801 26271 catalog_manager.cc:4830] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): tablet 2b50fc58a5384c6b8d2c78b88e0b3d95 (table rebalance_test_table_16 [id=f1e711a462c54504b4fe5c866615f824]) successfully deleted
I20240502 14:14:52.996913 30018 raft_consensus.cc:2229] T 5d8ed238257d46b9b78f0f8a19f2adba P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:52.997162 30018 raft_consensus.cc:2258] T 5d8ed238257d46b9b78f0f8a19f2adba P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.997956 30018 ts_tablet_manager.cc:1875] T 5d8ed238257d46b9b78f0f8a19f2adba P ba187e38a2754d60a12de9ac4856430a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:53.004309 29244 leader_election.cc:302] T 253449163c65471d8736b92b25da0f92 P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c1100e16b3b9495ca61d4f95e89b6a89; no voters: 1cb67158133e4648a6eceba32d4dbba3, ba187e38a2754d60a12de9ac4856430a
I20240502 14:14:53.004809 29872 raft_consensus.cc:2726] T 253449163c65471d8736b92b25da0f92 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:14:53.006188 30102 tablet_bootstrap.cc:492] T 83dd34749ff8496483365b30a86db71f P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:14:53.006358 30102 ts_tablet_manager.cc:1367] T 83dd34749ff8496483365b30a86db71f P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.146s user 0.005s sys 0.003s
I20240502 14:14:53.007236 30102 raft_consensus.cc:348] T 83dd34749ff8496483365b30a86db71f P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:53.007448 30102 raft_consensus.cc:374] T 83dd34749ff8496483365b30a86db71f P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:53.007544 30102 raft_consensus.cc:729] T 83dd34749ff8496483365b30a86db71f P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:14:53.007782 30102 consensus_queue.cc:260] T 83dd34749ff8496483365b30a86db71f P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:53.008378 30102 ts_tablet_manager.cc:1398] T 83dd34749ff8496483365b30a86db71f P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:14:53.012307 30018 ts_tablet_manager.cc:1888] T 5d8ed238257d46b9b78f0f8a19f2adba P ba187e38a2754d60a12de9ac4856430a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:53.012627 30018 log.cc:1196] T 5d8ed238257d46b9b78f0f8a19f2adba P ba187e38a2754d60a12de9ac4856430a: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/5d8ed238257d46b9b78f0f8a19f2adba
I20240502 14:14:53.013358 30018 ts_tablet_manager.cc:1909] T 5d8ed238257d46b9b78f0f8a19f2adba P ba187e38a2754d60a12de9ac4856430a: Deleting consensus metadata
I20240502 14:14:53.015676 26271 catalog_manager.cc:4830] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): tablet 5d8ed238257d46b9b78f0f8a19f2adba (table rebalance_test_table_16 [id=f1e711a462c54504b4fe5c866615f824]) successfully deleted
I20240502 14:14:53.024029 26427 raft_consensus.cc:1261] T 825ae3e90e214a6f98394765edf6ff2d P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:53.024936 29718 consensus_queue.cc:1035] T 825ae3e90e214a6f98394765edf6ff2d P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:53.048106 26818 raft_consensus.cc:1261] T 825ae3e90e214a6f98394765edf6ff2d P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:53.049012 29718 consensus_queue.cc:1035] T 825ae3e90e214a6f98394765edf6ff2d P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:53.053982 29645 raft_consensus.cc:2932] T 825ae3e90e214a6f98394765edf6ff2d P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:14:53.054584 26818 raft_consensus.cc:2932] T 825ae3e90e214a6f98394765edf6ff2d P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:14:53.065367 26280 catalog_manager.cc:5484] T 825ae3e90e214a6f98394765edf6ff2d P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: config changed from index -1 to 4, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:14:53.065560 26427 raft_consensus.cc:2932] T 825ae3e90e214a6f98394765edf6ff2d P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:14:53.073520 30118 tablet_bootstrap.cc:492] T 0fc2e6c74dc34a45ae1ec46608642880 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:53.074057 30118 tablet_bootstrap.cc:492] T 0fc2e6c74dc34a45ae1ec46608642880 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap complete.
I20240502 14:14:53.074357 30118 ts_tablet_manager.cc:1367] T 0fc2e6c74dc34a45ae1ec46608642880 P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.091s user 0.054s sys 0.008s
I20240502 14:14:53.075093 30118 raft_consensus.cc:348] T 0fc2e6c74dc34a45ae1ec46608642880 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:53.075408 30118 raft_consensus.cc:729] T 0fc2e6c74dc34a45ae1ec46608642880 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:14:53.075731 30118 consensus_queue.cc:260] T 0fc2e6c74dc34a45ae1ec46608642880 P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:53.076366 30118 ts_tablet_manager.cc:1398] T 0fc2e6c74dc34a45ae1ec46608642880 P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20240502 14:14:53.077273 26447 tablet_copy_service.cc:342] P 89af7f423cf44717a08d6b142f0ed49d: Request end of tablet copy session 1cb67158133e4648a6eceba32d4dbba3-0fc2e6c74dc34a45ae1ec46608642880 received from {username='slave'} at 127.19.94.197:56601
I20240502 14:14:53.077454 26447 tablet_copy_service.cc:433] P 89af7f423cf44717a08d6b142f0ed49d: ending tablet copy session 1cb67158133e4648a6eceba32d4dbba3-0fc2e6c74dc34a45ae1ec46608642880 on tablet 0fc2e6c74dc34a45ae1ec46608642880 with peer 1cb67158133e4648a6eceba32d4dbba3
I20240502 14:14:53.090656 30089 raft_consensus.cc:482] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:53.090868 30089 raft_consensus.cc:504] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:53.091678 30089 leader_election.cc:288] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:14:53.092612 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83dd34749ff8496483365b30a86db71f" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:14:53.092885 29443 raft_consensus.cc:2454] T 83dd34749ff8496483365b30a86db71f P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:14:53.093093 29174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83dd34749ff8496483365b30a86db71f" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:14:53.093328 29174 raft_consensus.cc:2454] T 83dd34749ff8496483365b30a86db71f P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:14:53.093611 29511 leader_election.cc:302] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, 68a3c2b17ed94e06aba66ed4178132ab; no voters:
I20240502 14:14:53.094202 30033 raft_consensus.cc:2781] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:53.094324 30033 raft_consensus.cc:482] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:53.094442 30033 raft_consensus.cc:3037] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:53.097785 30033 raft_consensus.cc:504] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
W20240502 14:14:53.099078 26494 consensus_peers.cc:485] T 825ae3e90e214a6f98394765edf6ff2d P ba187e38a2754d60a12de9ac4856430a -> Peer 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097): Couldn't send request to peer 4092902b3e674857a0147f1b4f2cdb86. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 825ae3e90e214a6f98394765edf6ff2d. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:53.099027 29174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83dd34749ff8496483365b30a86db71f" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:14:53.099242 29174 raft_consensus.cc:3037] T 83dd34749ff8496483365b30a86db71f P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:53.100889 30033 leader_election.cc:288] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Requested vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:14:53.101526 29174 raft_consensus.cc:2454] T 83dd34749ff8496483365b30a86db71f P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:14:53.102181 29510 leader_election.cc:302] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 4092902b3e674857a0147f1b4f2cdb86; no voters:
I20240502 14:14:53.103394 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83dd34749ff8496483365b30a86db71f" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
I20240502 14:14:53.103628 29443 raft_consensus.cc:3037] T 83dd34749ff8496483365b30a86db71f P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:53.105867 29443 raft_consensus.cc:2454] T 83dd34749ff8496483365b30a86db71f P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:14:53.106422 30033 raft_consensus.cc:2781] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:53.106659 30033 raft_consensus.cc:686] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Becoming Leader. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Running, Role: LEADER
I20240502 14:14:53.107009 30033 consensus_queue.cc:237] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:53.112210 26280 catalog_manager.cc:5484] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200). New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:14:53.123358 29872 raft_consensus.cc:482] T 56285a2fdcdd47e9b9c81b4b8993c4aa P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:53.123641 29872 raft_consensus.cc:504] T 56285a2fdcdd47e9b9c81b4b8993c4aa P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:53.124555 29872 leader_election.cc:288] T 56285a2fdcdd47e9b9c81b4b8993c4aa P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:14:53.125555 26427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56285a2fdcdd47e9b9c81b4b8993c4aa" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:14:53.125628 26557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56285a2fdcdd47e9b9c81b4b8993c4aa" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:53.125825 26427 raft_consensus.cc:2454] T 56285a2fdcdd47e9b9c81b4b8993c4aa P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 0.
I20240502 14:14:53.125977 26557 raft_consensus.cc:2454] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 0.
I20240502 14:14:53.126426 29244 leader_election.cc:302] T 56285a2fdcdd47e9b9c81b4b8993c4aa P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:14:53.127151 29872 raft_consensus.cc:2781] T 56285a2fdcdd47e9b9c81b4b8993c4aa P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:53.127282 29872 raft_consensus.cc:482] T 56285a2fdcdd47e9b9c81b4b8993c4aa P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:53.127372 29872 raft_consensus.cc:3037] T 56285a2fdcdd47e9b9c81b4b8993c4aa P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:53.131021 29872 raft_consensus.cc:504] T 56285a2fdcdd47e9b9c81b4b8993c4aa P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:53.132351 29872 leader_election.cc:288] T 56285a2fdcdd47e9b9c81b4b8993c4aa P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 election: Requested vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:14:53.132885 26423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56285a2fdcdd47e9b9c81b4b8993c4aa" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:14:53.133215 26423 raft_consensus.cc:3037] T 56285a2fdcdd47e9b9c81b4b8993c4aa P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:53.139971 26557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56285a2fdcdd47e9b9c81b4b8993c4aa" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:14:53.140239 26557 raft_consensus.cc:3037] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:53.142165 26423 raft_consensus.cc:2454] T 56285a2fdcdd47e9b9c81b4b8993c4aa P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 1.
I20240502 14:14:53.142764 29244 leader_election.cc:302] T 56285a2fdcdd47e9b9c81b4b8993c4aa P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:14:53.143231 29872 raft_consensus.cc:2781] T 56285a2fdcdd47e9b9c81b4b8993c4aa P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:53.143385 29872 raft_consensus.cc:686] T 56285a2fdcdd47e9b9c81b4b8993c4aa P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 LEADER]: Becoming Leader. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Running, Role: LEADER
I20240502 14:14:53.143684 26557 raft_consensus.cc:2454] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 1.
I20240502 14:14:53.143780 29872 consensus_queue.cc:237] T 56285a2fdcdd47e9b9c81b4b8993c4aa P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:53.153810 26280 catalog_manager.cc:5484] T 56285a2fdcdd47e9b9c81b4b8993c4aa P c1100e16b3b9495ca61d4f95e89b6a89 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198). New cstate: current_term: 1 leader_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:14:53.161317 26280 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:14:53.173664 29174 raft_consensus.cc:1261] T 1a3e938ab64e4fc09462c2dbfcee015e P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer c1100e16b3b9495ca61d4f95e89b6a89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:53.174541 29872 consensus_queue.cc:1035] T 1a3e938ab64e4fc09462c2dbfcee015e P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:53.177019 30155 ts_tablet_manager.cc:902] T bb3e4ec75888419aa4e065cf5513ae5b P c1100e16b3b9495ca61d4f95e89b6a89: Initiating tablet copy from peer 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:14:53.182777 29574 raft_consensus.cc:1261] T 1a3e938ab64e4fc09462c2dbfcee015e P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Refusing update from remote peer c1100e16b3b9495ca61d4f95e89b6a89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:53.183609 29872 consensus_queue.cc:1035] T 1a3e938ab64e4fc09462c2dbfcee015e P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:53.235896 26280 catalog_manager.cc:5576] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097) does not have the latest schema for tablet 1a3e938ab64e4fc09462c2dbfcee015e (table rebalancer_dynamic_table [id=58dd440e1ee2463cb0df003412402dcd]). Expected version 1 got 0
I20240502 14:14:53.248090 30161 mvcc.cc:205] Tried to move back new op lower bound from 7023244464820887552 to 7023244463808532480. Current Snapshot: MvccSnapshot[applied={T|T < 7023244464820887552}]
I20240502 14:14:53.276837 26280 catalog_manager.cc:5576] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097) does not have the latest schema for tablet 1a3e938ab64e4fc09462c2dbfcee015e (table rebalancer_dynamic_table [id=58dd440e1ee2463cb0df003412402dcd]). Expected version 1 got 0
I20240502 14:14:53.278796 30163 tablet.cc:1722] T 1a3e938ab64e4fc09462c2dbfcee015e P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:53.279587 30163 tablet.cc:1620] T 1a3e938ab64e4fc09462c2dbfcee015e P c1100e16b3b9495ca61d4f95e89b6a89: MemRowSet was empty: no flush needed.
I20240502 14:14:53.273375 30164 tablet.cc:1722] T 1a3e938ab64e4fc09462c2dbfcee015e P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:53.280798 30164 tablet.cc:1620] T 1a3e938ab64e4fc09462c2dbfcee015e P 4092902b3e674857a0147f1b4f2cdb86: MemRowSet was empty: no flush needed.
I20240502 14:14:53.282702 26280 catalog_manager.cc:5576] TS 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715) does not have the latest schema for tablet 1a3e938ab64e4fc09462c2dbfcee015e (table rebalancer_dynamic_table [id=58dd440e1ee2463cb0df003412402dcd]). Expected version 1 got 0
I20240502 14:14:53.302634 26280 catalog_manager.cc:5576] TS 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715) does not have the latest schema for tablet 1a3e938ab64e4fc09462c2dbfcee015e (table rebalancer_dynamic_table [id=58dd440e1ee2463cb0df003412402dcd]). Expected version 1 got 0
I20240502 14:14:53.300024 30167 tablet.cc:1722] T 1a3e938ab64e4fc09462c2dbfcee015e P 1cb67158133e4648a6eceba32d4dbba3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:53.310169 30167 tablet.cc:1620] T 1a3e938ab64e4fc09462c2dbfcee015e P 1cb67158133e4648a6eceba32d4dbba3: MemRowSet was empty: no flush needed.
I20240502 14:14:53.324379 26280 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=58dd440e1ee2463cb0df003412402dcd] alter complete (version 1)
I20240502 14:14:53.353302 29176 raft_consensus.cc:1203] T 0fc2e6c74dc34a45ae1ec46608642880 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:53.364719 30155 tablet_copy_client.cc:308] T bb3e4ec75888419aa4e065cf5513ae5b P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.193:37985
I20240502 14:14:53.365818 26447 tablet_copy_service.cc:140] P 89af7f423cf44717a08d6b142f0ed49d: Received BeginTabletCopySession request for tablet bb3e4ec75888419aa4e065cf5513ae5b from peer c1100e16b3b9495ca61d4f95e89b6a89 ({username='slave'} at 127.19.94.198:52561)
I20240502 14:14:53.366029 26447 tablet_copy_service.cc:161] P 89af7f423cf44717a08d6b142f0ed49d: Beginning new tablet copy session on tablet bb3e4ec75888419aa4e065cf5513ae5b from peer c1100e16b3b9495ca61d4f95e89b6a89 at {username='slave'} at 127.19.94.198:52561: session id = c1100e16b3b9495ca61d4f95e89b6a89-bb3e4ec75888419aa4e065cf5513ae5b
I20240502 14:14:53.368489 26447 tablet_copy_source_session.cc:205] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:53.370901 30155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb3e4ec75888419aa4e065cf5513ae5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:53.376904 30155 tablet_copy_client.cc:783] T bb3e4ec75888419aa4e065cf5513ae5b P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:53.376818 26689 consensus_queue.cc:237] T f7773bd35a1941c690e620753f42019b P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:53.377249 30155 tablet_copy_client.cc:647] T bb3e4ec75888419aa4e065cf5513ae5b P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:53.379577 30155 tablet_copy_client.cc:523] T bb3e4ec75888419aa4e065cf5513ae5b P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:53.380729 26557 raft_consensus.cc:1261] T f7773bd35a1941c690e620753f42019b P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:14:53.380896 26625 consensus_peers.cc:485] T f7773bd35a1941c690e620753f42019b P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f7773bd35a1941c690e620753f42019b. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:53.381069 26423 raft_consensus.cc:1261] T f7773bd35a1941c690e620753f42019b P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:53.381574 30010 consensus_queue.cc:1035] T f7773bd35a1941c690e620753f42019b P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:53.381923 30177 consensus_queue.cc:1035] T f7773bd35a1941c690e620753f42019b P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:53.383641 30155 tablet_bootstrap.cc:492] T bb3e4ec75888419aa4e065cf5513ae5b P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:14:53.385226 29918 consensus_queue.cc:1035] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:53.390286 29227 raft_consensus.cc:2932] T f7773bd35a1941c690e620753f42019b P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:14:53.391197 26558 raft_consensus.cc:2932] T f7773bd35a1941c690e620753f42019b P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:14:53.397855 26423 raft_consensus.cc:2932] T f7773bd35a1941c690e620753f42019b P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:14:53.403306 26276 catalog_manager.cc:5484] T f7773bd35a1941c690e620753f42019b P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: config changed from index -1 to 4, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:14:53.418078 29918 consensus_queue.cc:1035] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:53.441720 26280 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240502 14:14:53.441677 26276 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:14:53.441949 26280 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240502 14:14:53.451615 26280 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet 001f99a64fb94ec4ba7abacc41953cb7
I20240502 14:14:53.452132 26280 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet d1b08a572d684a82b85c5f9f461435ab
I20240502 14:14:53.452492 26280 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 4 replicas of tablet f7773bd35a1941c690e620753f42019b
I20240502 14:14:53.453974 26407 tablet_service.cc:1514] Processing DeleteTablet for tablet f7773bd35a1941c690e620753f42019b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:53 UTC) from {username='slave'} at 127.0.0.1:37422
I20240502 14:14:53.454258 30017 tablet_replica.cc:330] T f7773bd35a1941c690e620753f42019b P 89af7f423cf44717a08d6b142f0ed49d: stopping tablet replica
I20240502 14:14:53.454484 30017 raft_consensus.cc:2229] T f7773bd35a1941c690e620753f42019b P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:53.454742 30017 raft_consensus.cc:2258] T f7773bd35a1941c690e620753f42019b P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:53.455173 26667 tablet_service.cc:1514] Processing DeleteTablet for tablet 001f99a64fb94ec4ba7abacc41953cb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:53 UTC) from {username='slave'} at 127.0.0.1:34150
I20240502 14:14:53.456053 26669 tablet_service.cc:1514] Processing DeleteTablet for tablet d1b08a572d684a82b85c5f9f461435ab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:53 UTC) from {username='slave'} at 127.0.0.1:34150
I20240502 14:14:53.456154 30017 ts_tablet_manager.cc:1875] T f7773bd35a1941c690e620753f42019b P 89af7f423cf44717a08d6b142f0ed49d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:53.456185 26406 tablet_service.cc:1514] Processing DeleteTablet for tablet 001f99a64fb94ec4ba7abacc41953cb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:53 UTC) from {username='slave'} at 127.0.0.1:37422
I20240502 14:14:53.456317 26668 tablet_service.cc:1514] Processing DeleteTablet for tablet f7773bd35a1941c690e620753f42019b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:53 UTC) from {username='slave'} at 127.0.0.1:34150
I20240502 14:14:53.456518 26797 tablet_service.cc:1514] Processing DeleteTablet for tablet d1b08a572d684a82b85c5f9f461435ab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:53 UTC) from {username='slave'} at 127.0.0.1:56010
I20240502 14:14:53.456586 30020 tablet_replica.cc:330] T 001f99a64fb94ec4ba7abacc41953cb7 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: stopping tablet replica
I20240502 14:14:53.456811 30020 raft_consensus.cc:2229] T 001f99a64fb94ec4ba7abacc41953cb7 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:53.457082 30020 raft_consensus.cc:2258] T 001f99a64fb94ec4ba7abacc41953cb7 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:53.457129 29422 tablet_service.cc:1514] Processing DeleteTablet for tablet f7773bd35a1941c690e620753f42019b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:53 UTC) from {username='slave'} at 127.0.0.1:40652
I20240502 14:14:53.457882 30020 ts_tablet_manager.cc:1875] T 001f99a64fb94ec4ba7abacc41953cb7 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:53.458005 30019 tablet_replica.cc:330] T d1b08a572d684a82b85c5f9f461435ab P e4cfb8fc5a474177af6848a6efa13a27: stopping tablet replica
I20240502 14:14:53.458227 30019 raft_consensus.cc:2229] T d1b08a572d684a82b85c5f9f461435ab P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:53.458499 30019 raft_consensus.cc:2258] T d1b08a572d684a82b85c5f9f461435ab P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:53.459215 30019 ts_tablet_manager.cc:1875] T d1b08a572d684a82b85c5f9f461435ab P e4cfb8fc5a474177af6848a6efa13a27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:53.462265 30163 tablet.cc:1722] T 1a3e938ab64e4fc09462c2dbfcee015e P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:53.462826 30163 tablet.cc:1620] T 1a3e938ab64e4fc09462c2dbfcee015e P c1100e16b3b9495ca61d4f95e89b6a89: MemRowSet was empty: no flush needed.
I20240502 14:14:53.466382 26536 tablet_service.cc:1514] Processing DeleteTablet for tablet 001f99a64fb94ec4ba7abacc41953cb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:53 UTC) from {username='slave'} at 127.0.0.1:51696
I20240502 14:14:53.466502 26538 tablet_service.cc:1514] Processing DeleteTablet for tablet d1b08a572d684a82b85c5f9f461435ab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:53 UTC) from {username='slave'} at 127.0.0.1:51696
I20240502 14:14:53.466761 26537 tablet_service.cc:1514] Processing DeleteTablet for tablet f7773bd35a1941c690e620753f42019b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:53 UTC) from {username='slave'} at 127.0.0.1:51696
I20240502 14:14:53.466990 30018 tablet_replica.cc:330] T 001f99a64fb94ec4ba7abacc41953cb7 P ba187e38a2754d60a12de9ac4856430a: stopping tablet replica
I20240502 14:14:53.467224 30018 raft_consensus.cc:2229] T 001f99a64fb94ec4ba7abacc41953cb7 P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:53.467592 30018 raft_consensus.cc:2258] T 001f99a64fb94ec4ba7abacc41953cb7 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:53.468418 30018 ts_tablet_manager.cc:1875] T 001f99a64fb94ec4ba7abacc41953cb7 P ba187e38a2754d60a12de9ac4856430a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:53.469957 30017 ts_tablet_manager.cc:1888] T f7773bd35a1941c690e620753f42019b P 89af7f423cf44717a08d6b142f0ed49d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:53.470111 30017 log.cc:1196] T f7773bd35a1941c690e620753f42019b P 89af7f423cf44717a08d6b142f0ed49d: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/f7773bd35a1941c690e620753f42019b
I20240502 14:14:53.470556 30017 ts_tablet_manager.cc:1909] T f7773bd35a1941c690e620753f42019b P 89af7f423cf44717a08d6b142f0ed49d: Deleting consensus metadata
I20240502 14:14:53.472622 26271 catalog_manager.cc:4830] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): tablet f7773bd35a1941c690e620753f42019b (table rebalance_test_table_18 [id=5b975fa2c29340a4a7018f3ddff103a0]) successfully deleted
I20240502 14:14:53.472997 30017 tablet_replica.cc:330] T 001f99a64fb94ec4ba7abacc41953cb7 P 89af7f423cf44717a08d6b142f0ed49d: stopping tablet replica
I20240502 14:14:53.473237 30017 raft_consensus.cc:2229] T 001f99a64fb94ec4ba7abacc41953cb7 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:53.473582 30017 raft_consensus.cc:2258] T 001f99a64fb94ec4ba7abacc41953cb7 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:53.475286 30017 ts_tablet_manager.cc:1875] T 001f99a64fb94ec4ba7abacc41953cb7 P 89af7f423cf44717a08d6b142f0ed49d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:53.476832 30167 tablet.cc:1722] T 1a3e938ab64e4fc09462c2dbfcee015e P 1cb67158133e4648a6eceba32d4dbba3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:53.477398 30167 tablet.cc:1620] T 1a3e938ab64e4fc09462c2dbfcee015e P 1cb67158133e4648a6eceba32d4dbba3: MemRowSet was empty: no flush needed.
I20240502 14:14:53.478955 30164 tablet.cc:1722] T 1a3e938ab64e4fc09462c2dbfcee015e P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:53.479481 30164 tablet.cc:1620] T 1a3e938ab64e4fc09462c2dbfcee015e P 4092902b3e674857a0147f1b4f2cdb86: MemRowSet was empty: no flush needed.
I20240502 14:14:53.490734 30019 ts_tablet_manager.cc:1888] T d1b08a572d684a82b85c5f9f461435ab P e4cfb8fc5a474177af6848a6efa13a27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:53.491135 26283 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=58dd440e1ee2463cb0df003412402dcd] alter complete (version 2)
I20240502 14:14:53.495118 30019 log.cc:1196] T d1b08a572d684a82b85c5f9f461435ab P e4cfb8fc5a474177af6848a6efa13a27: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/wals/d1b08a572d684a82b85c5f9f461435ab
I20240502 14:14:53.495939 30019 ts_tablet_manager.cc:1909] T d1b08a572d684a82b85c5f9f461435ab P e4cfb8fc5a474177af6848a6efa13a27: Deleting consensus metadata
I20240502 14:14:53.497737 26272 catalog_manager.cc:4830] TS e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): tablet d1b08a572d684a82b85c5f9f461435ab (table rebalance_test_table_18 [id=5b975fa2c29340a4a7018f3ddff103a0]) successfully deleted
I20240502 14:14:53.498167 30020 ts_tablet_manager.cc:1888] T 001f99a64fb94ec4ba7abacc41953cb7 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:53.498342 30020 log.cc:1196] T 001f99a64fb94ec4ba7abacc41953cb7 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/wals/001f99a64fb94ec4ba7abacc41953cb7
I20240502 14:14:53.498808 30020 ts_tablet_manager.cc:1909] T 001f99a64fb94ec4ba7abacc41953cb7 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting consensus metadata
I20240502 14:14:53.499402 30018 ts_tablet_manager.cc:1888] T 001f99a64fb94ec4ba7abacc41953cb7 P ba187e38a2754d60a12de9ac4856430a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:53.499701 30018 log.cc:1196] T 001f99a64fb94ec4ba7abacc41953cb7 P ba187e38a2754d60a12de9ac4856430a: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/001f99a64fb94ec4ba7abacc41953cb7
I20240502 14:14:53.501344 26272 catalog_manager.cc:4830] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): tablet 001f99a64fb94ec4ba7abacc41953cb7 (table rebalance_test_table_18 [id=5b975fa2c29340a4a7018f3ddff103a0]) successfully deleted
I20240502 14:14:53.502357 30018 ts_tablet_manager.cc:1909] T 001f99a64fb94ec4ba7abacc41953cb7 P ba187e38a2754d60a12de9ac4856430a: Deleting consensus metadata
I20240502 14:14:53.503198 30017 ts_tablet_manager.cc:1888] T 001f99a64fb94ec4ba7abacc41953cb7 P 89af7f423cf44717a08d6b142f0ed49d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
W20240502 14:14:53.503203 26272 catalog_manager.cc:4797] TS 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): delete failed for tablet f7773bd35a1941c690e620753f42019b because the tablet was not found. No further retry: Not found: Tablet not found: f7773bd35a1941c690e620753f42019b
I20240502 14:14:53.503580 30017 log.cc:1196] T 001f99a64fb94ec4ba7abacc41953cb7 P 89af7f423cf44717a08d6b142f0ed49d: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/001f99a64fb94ec4ba7abacc41953cb7
I20240502 14:14:53.503610 30018 tablet_replica.cc:330] T d1b08a572d684a82b85c5f9f461435ab P ba187e38a2754d60a12de9ac4856430a: stopping tablet replica
I20240502 14:14:53.503891 30020 tablet_replica.cc:330] T d1b08a572d684a82b85c5f9f461435ab P 69ab78b2a1b74d2bbf2cdeafa0a4af23: stopping tablet replica
I20240502 14:14:53.503887 26271 catalog_manager.cc:4830] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): tablet 001f99a64fb94ec4ba7abacc41953cb7 (table rebalance_test_table_18 [id=5b975fa2c29340a4a7018f3ddff103a0]) successfully deleted
I20240502 14:14:53.504248 30017 ts_tablet_manager.cc:1909] T 001f99a64fb94ec4ba7abacc41953cb7 P 89af7f423cf44717a08d6b142f0ed49d: Deleting consensus metadata
I20240502 14:14:53.504242 30020 raft_consensus.cc:2229] T d1b08a572d684a82b85c5f9f461435ab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:53.504755 30020 raft_consensus.cc:2258] T d1b08a572d684a82b85c5f9f461435ab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:53.505046 30018 raft_consensus.cc:2229] T d1b08a572d684a82b85c5f9f461435ab P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:53.505872 26271 catalog_manager.cc:4830] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): tablet 001f99a64fb94ec4ba7abacc41953cb7 (table rebalance_test_table_18 [id=5b975fa2c29340a4a7018f3ddff103a0]) successfully deleted
I20240502 14:14:53.506299 30018 raft_consensus.cc:2258] T d1b08a572d684a82b85c5f9f461435ab P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:53.507112 30018 ts_tablet_manager.cc:1875] T d1b08a572d684a82b85c5f9f461435ab P ba187e38a2754d60a12de9ac4856430a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:53.507652 30020 ts_tablet_manager.cc:1875] T d1b08a572d684a82b85c5f9f461435ab P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:53.519006 30018 ts_tablet_manager.cc:1888] T d1b08a572d684a82b85c5f9f461435ab P ba187e38a2754d60a12de9ac4856430a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:53.528129 30020 ts_tablet_manager.cc:1888] T d1b08a572d684a82b85c5f9f461435ab P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:53.528299 30020 log.cc:1196] T d1b08a572d684a82b85c5f9f461435ab P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/wals/d1b08a572d684a82b85c5f9f461435ab
I20240502 14:14:53.528829 30020 ts_tablet_manager.cc:1909] T d1b08a572d684a82b85c5f9f461435ab P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting consensus metadata
I20240502 14:14:53.529023 30018 log.cc:1196] T d1b08a572d684a82b85c5f9f461435ab P ba187e38a2754d60a12de9ac4856430a: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/d1b08a572d684a82b85c5f9f461435ab
I20240502 14:14:53.529544 30018 ts_tablet_manager.cc:1909] T d1b08a572d684a82b85c5f9f461435ab P ba187e38a2754d60a12de9ac4856430a: Deleting consensus metadata
I20240502 14:14:53.530752 26272 catalog_manager.cc:4830] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): tablet d1b08a572d684a82b85c5f9f461435ab (table rebalance_test_table_18 [id=5b975fa2c29340a4a7018f3ddff103a0]) successfully deleted
I20240502 14:14:53.532081 26271 catalog_manager.cc:4830] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): tablet d1b08a572d684a82b85c5f9f461435ab (table rebalance_test_table_18 [id=5b975fa2c29340a4a7018f3ddff103a0]) successfully deleted
I20240502 14:14:53.533483 30020 tablet_replica.cc:330] T f7773bd35a1941c690e620753f42019b P 69ab78b2a1b74d2bbf2cdeafa0a4af23: stopping tablet replica
I20240502 14:14:53.533743 30018 tablet_replica.cc:330] T f7773bd35a1941c690e620753f42019b P ba187e38a2754d60a12de9ac4856430a: stopping tablet replica
I20240502 14:14:53.534049 30020 raft_consensus.cc:2229] T f7773bd35a1941c690e620753f42019b P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:53.534376 30018 raft_consensus.cc:2229] T f7773bd35a1941c690e620753f42019b P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:53.534965 30020 raft_consensus.cc:2258] T f7773bd35a1941c690e620753f42019b P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:53.535387 30018 raft_consensus.cc:2258] T f7773bd35a1941c690e620753f42019b P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:53.536345 30018 ts_tablet_manager.cc:1875] T f7773bd35a1941c690e620753f42019b P ba187e38a2754d60a12de9ac4856430a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:53.537372 30020 ts_tablet_manager.cc:1875] T f7773bd35a1941c690e620753f42019b P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:53.541393 30155 tablet_bootstrap.cc:492] T bb3e4ec75888419aa4e065cf5513ae5b P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:53.541930 30155 tablet_bootstrap.cc:492] T bb3e4ec75888419aa4e065cf5513ae5b P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap complete.
I20240502 14:14:53.542263 30155 ts_tablet_manager.cc:1367] T bb3e4ec75888419aa4e065cf5513ae5b P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.159s user 0.068s sys 0.026s
I20240502 14:14:53.543181 30155 raft_consensus.cc:348] T bb3e4ec75888419aa4e065cf5513ae5b P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:53.543489 30155 raft_consensus.cc:729] T bb3e4ec75888419aa4e065cf5513ae5b P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:14:53.543771 30155 consensus_queue.cc:260] T bb3e4ec75888419aa4e065cf5513ae5b P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:53.544428 30155 ts_tablet_manager.cc:1398] T bb3e4ec75888419aa4e065cf5513ae5b P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:14:53.545459 26447 tablet_copy_service.cc:342] P 89af7f423cf44717a08d6b142f0ed49d: Request end of tablet copy session c1100e16b3b9495ca61d4f95e89b6a89-bb3e4ec75888419aa4e065cf5513ae5b received from {username='slave'} at 127.19.94.198:52561
I20240502 14:14:53.545605 26447 tablet_copy_service.cc:433] P 89af7f423cf44717a08d6b142f0ed49d: ending tablet copy session c1100e16b3b9495ca61d4f95e89b6a89-bb3e4ec75888419aa4e065cf5513ae5b on tablet bb3e4ec75888419aa4e065cf5513ae5b with peer c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:14:53.551821 30018 ts_tablet_manager.cc:1888] T f7773bd35a1941c690e620753f42019b P ba187e38a2754d60a12de9ac4856430a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:53.552098 30018 log.cc:1196] T f7773bd35a1941c690e620753f42019b P ba187e38a2754d60a12de9ac4856430a: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/f7773bd35a1941c690e620753f42019b
I20240502 14:14:53.552703 30018 ts_tablet_manager.cc:1909] T f7773bd35a1941c690e620753f42019b P ba187e38a2754d60a12de9ac4856430a: Deleting consensus metadata
I20240502 14:14:53.552950 30020 ts_tablet_manager.cc:1888] T f7773bd35a1941c690e620753f42019b P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:53.553189 30020 log.cc:1196] T f7773bd35a1941c690e620753f42019b P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/wals/f7773bd35a1941c690e620753f42019b
I20240502 14:14:53.553874 30020 ts_tablet_manager.cc:1909] T f7773bd35a1941c690e620753f42019b P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting consensus metadata
I20240502 14:14:53.553989 26271 catalog_manager.cc:4830] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): tablet f7773bd35a1941c690e620753f42019b (table rebalance_test_table_18 [id=5b975fa2c29340a4a7018f3ddff103a0]) successfully deleted
I20240502 14:14:53.555218 26272 catalog_manager.cc:4830] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): tablet f7773bd35a1941c690e620753f42019b (table rebalance_test_table_18 [id=5b975fa2c29340a4a7018f3ddff103a0]) successfully deleted
I20240502 14:14:53.555526 30089 consensus_queue.cc:1035] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:53.565438 30089 consensus_queue.cc:1035] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:53.590863 29309 raft_consensus.cc:1203] T bb3e4ec75888419aa4e065cf5513ae5b P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:53.591099 26818 consensus_queue.cc:237] T 981e25d29abc44299dc115d6df835c58 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:53.593955 26689 raft_consensus.cc:1261] T 981e25d29abc44299dc115d6df835c58 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:53.594679 26423 raft_consensus.cc:1261] T 981e25d29abc44299dc115d6df835c58 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:53.594630 29791 consensus_queue.cc:1035] T 981e25d29abc44299dc115d6df835c58 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:53.595003 30127 consensus_queue.cc:1035] T 56285a2fdcdd47e9b9c81b4b8993c4aa P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:53.595539 26754 consensus_peers.cc:485] T 981e25d29abc44299dc115d6df835c58 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): Couldn't send request to peer 1cb67158133e4648a6eceba32d4dbba3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 981e25d29abc44299dc115d6df835c58. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:53.595953 29791 consensus_queue.cc:1035] T 981e25d29abc44299dc115d6df835c58 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:53.598373 26279 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:14:53.615909 30163 tablet.cc:1722] T 1a3e938ab64e4fc09462c2dbfcee015e P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:53.622871 30164 tablet.cc:1722] T 1a3e938ab64e4fc09462c2dbfcee015e P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:53.632547 30167 tablet.cc:1722] T 1a3e938ab64e4fc09462c2dbfcee015e P 1cb67158133e4648a6eceba32d4dbba3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:53.634505 29872 consensus_queue.cc:1035] T 56285a2fdcdd47e9b9c81b4b8993c4aa P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:53.644711 26283 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table_new_name [id=58dd440e1ee2463cb0df003412402dcd] alter complete (version 3)
I20240502 14:14:53.646225 30209 ts_tablet_manager.cc:902] T 825ae3e90e214a6f98394765edf6ff2d P 4092902b3e674857a0147f1b4f2cdb86: Initiating tablet copy from peer ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:14:53.650034 29226 raft_consensus.cc:2932] T 981e25d29abc44299dc115d6df835c58 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:14:53.650745 26423 raft_consensus.cc:2932] T 981e25d29abc44299dc115d6df835c58 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:14:53.657866 26283 catalog_manager.cc:5484] T 981e25d29abc44299dc115d6df835c58 P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: config changed from index -1 to 4, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:14:53.672708 26689 raft_consensus.cc:2932] T 981e25d29abc44299dc115d6df835c58 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:14:53.684033 30209 tablet_copy_client.cc:308] T 825ae3e90e214a6f98394765edf6ff2d P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.194:36025
I20240502 14:14:53.708999 26578 tablet_copy_service.cc:140] P ba187e38a2754d60a12de9ac4856430a: Received BeginTabletCopySession request for tablet 825ae3e90e214a6f98394765edf6ff2d from peer 4092902b3e674857a0147f1b4f2cdb86 ({username='slave'} at 127.19.94.200:52883)
I20240502 14:14:53.709277 26578 tablet_copy_service.cc:161] P ba187e38a2754d60a12de9ac4856430a: Beginning new tablet copy session on tablet 825ae3e90e214a6f98394765edf6ff2d from peer 4092902b3e674857a0147f1b4f2cdb86 at {username='slave'} at 127.19.94.200:52883: session id = 4092902b3e674857a0147f1b4f2cdb86-825ae3e90e214a6f98394765edf6ff2d
I20240502 14:14:53.712172 26578 tablet_copy_source_session.cc:205] T 825ae3e90e214a6f98394765edf6ff2d P ba187e38a2754d60a12de9ac4856430a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:53.714848 30209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 825ae3e90e214a6f98394765edf6ff2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:53.717124 26283 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_extra_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:53.722128 30209 tablet_copy_client.cc:783] T 825ae3e90e214a6f98394765edf6ff2d P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:53.722492 30209 tablet_copy_client.cc:647] T 825ae3e90e214a6f98394765edf6ff2d P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:53.725255 30209 tablet_copy_client.cc:523] T 825ae3e90e214a6f98394765edf6ff2d P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:53.728868 29422 tablet_service.cc:1467] Processing CreateTablet for tablet 4cea6495041642f58e1eb5d26fc10df5 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=77b7f84f16494751ad1f1e50fa6a8db0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:53.729358 29422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cea6495041642f58e1eb5d26fc10df5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:53.729449 30209 tablet_bootstrap.cc:492] T 825ae3e90e214a6f98394765edf6ff2d P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:14:53.729511 29287 tablet_service.cc:1467] Processing CreateTablet for tablet 4cea6495041642f58e1eb5d26fc10df5 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=77b7f84f16494751ad1f1e50fa6a8db0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:53.729938 29287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cea6495041642f58e1eb5d26fc10df5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:53.732856 26406 tablet_service.cc:1467] Processing CreateTablet for tablet 4cea6495041642f58e1eb5d26fc10df5 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=77b7f84f16494751ad1f1e50fa6a8db0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:53.733327 26406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cea6495041642f58e1eb5d26fc10df5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:53.740900 29423 tablet_service.cc:1467] Processing CreateTablet for tablet f94a6ca24bf748c1ac4f12df761d537d (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=77b7f84f16494751ad1f1e50fa6a8db0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:53.740303 29288 tablet_service.cc:1467] Processing CreateTablet for tablet f94a6ca24bf748c1ac4f12df761d537d (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=77b7f84f16494751ad1f1e50fa6a8db0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:53.742051 29288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f94a6ca24bf748c1ac4f12df761d537d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:53.742014 29156 tablet_service.cc:1467] Processing CreateTablet for tablet f94a6ca24bf748c1ac4f12df761d537d (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=77b7f84f16494751ad1f1e50fa6a8db0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:53.742431 29156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f94a6ca24bf748c1ac4f12df761d537d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:53.751379 26280 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:14:53.751658 26280 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:14:53.760919 29423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f94a6ca24bf748c1ac4f12df761d537d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:53.760939 29155 tablet_service.cc:1467] Processing CreateTablet for tablet 2d724f07a884486f9c5298ff585d62cb (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=77b7f84f16494751ad1f1e50fa6a8db0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:53.761396 29155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d724f07a884486f9c5298ff585d62cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:53.764869 29554 tablet_service.cc:1467] Processing CreateTablet for tablet 2d724f07a884486f9c5298ff585d62cb (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=77b7f84f16494751ad1f1e50fa6a8db0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:53.765374 29554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d724f07a884486f9c5298ff585d62cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:53.769346 26280 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet 1a3e938ab64e4fc09462c2dbfcee015e
I20240502 14:14:53.771224 29288 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a3e938ab64e4fc09462c2dbfcee015e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:53 UTC) from {username='slave'} at 127.0.0.1:59160
I20240502 14:14:53.772177 29154 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a3e938ab64e4fc09462c2dbfcee015e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:53 UTC) from {username='slave'} at 127.0.0.1:55202
I20240502 14:14:53.773070 26668 tablet_service.cc:1467] Processing CreateTablet for tablet 2d724f07a884486f9c5298ff585d62cb (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=77b7f84f16494751ad1f1e50fa6a8db0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:53.773543 26668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d724f07a884486f9c5298ff585d62cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:53.776603 29553 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a3e938ab64e4fc09462c2dbfcee015e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:53 UTC) from {username='slave'} at 127.0.0.1:36398
I20240502 14:14:53.797693 30229 tablet_bootstrap.cc:492] T 4cea6495041642f58e1eb5d26fc10df5 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:14:53.805958 30229 tablet_bootstrap.cc:654] T 4cea6495041642f58e1eb5d26fc10df5 P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:53.808406 30239 tablet_bootstrap.cc:492] T 2d724f07a884486f9c5298ff585d62cb P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:14:53.809479 30237 tablet_bootstrap.cc:492] T 2d724f07a884486f9c5298ff585d62cb P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:53.806882 30234 tablet_replica.cc:330] T 1a3e938ab64e4fc09462c2dbfcee015e P c1100e16b3b9495ca61d4f95e89b6a89: stopping tablet replica
I20240502 14:14:53.810216 30234 raft_consensus.cc:2229] T 1a3e938ab64e4fc09462c2dbfcee015e P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:53.810954 30234 raft_consensus.cc:2258] T 1a3e938ab64e4fc09462c2dbfcee015e P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:53.812790 30234 ts_tablet_manager.cc:1875] T 1a3e938ab64e4fc09462c2dbfcee015e P c1100e16b3b9495ca61d4f95e89b6a89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:53.813206 30237 tablet_bootstrap.cc:654] T 2d724f07a884486f9c5298ff585d62cb P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:53.814357 30233 tablet_bootstrap.cc:492] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:14:53.814781 30229 tablet_bootstrap.cc:492] T 4cea6495041642f58e1eb5d26fc10df5 P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:14:53.816792 30232 tablet_replica.cc:330] T 1a3e938ab64e4fc09462c2dbfcee015e P 1cb67158133e4648a6eceba32d4dbba3: stopping tablet replica
I20240502 14:14:53.817059 30232 raft_consensus.cc:2229] T 1a3e938ab64e4fc09462c2dbfcee015e P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:53.817256 30229 ts_tablet_manager.cc:1367] T 4cea6495041642f58e1eb5d26fc10df5 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.005s
I20240502 14:14:53.817333 30232 raft_consensus.cc:2258] T 1a3e938ab64e4fc09462c2dbfcee015e P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:53.817991 30231 tablet_bootstrap.cc:492] T f94a6ca24bf748c1ac4f12df761d537d P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:14:53.820209 30235 tablet_bootstrap.cc:492] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:53.820868 30236 tablet_replica.cc:330] T 1a3e938ab64e4fc09462c2dbfcee015e P 4092902b3e674857a0147f1b4f2cdb86: stopping tablet replica
I20240502 14:14:53.821118 30236 raft_consensus.cc:2229] T 1a3e938ab64e4fc09462c2dbfcee015e P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:53.821398 30236 raft_consensus.cc:2258] T 1a3e938ab64e4fc09462c2dbfcee015e P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:53.821671 30239 tablet_bootstrap.cc:654] T 2d724f07a884486f9c5298ff585d62cb P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:53.822543 30236 ts_tablet_manager.cc:1875] T 1a3e938ab64e4fc09462c2dbfcee015e P 4092902b3e674857a0147f1b4f2cdb86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:53.823650 30237 tablet_bootstrap.cc:492] T 2d724f07a884486f9c5298ff585d62cb P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:53.823863 30237 ts_tablet_manager.cc:1367] T 2d724f07a884486f9c5298ff585d62cb P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.004s
I20240502 14:14:53.824787 30237 raft_consensus.cc:348] T 2d724f07a884486f9c5298ff585d62cb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:53.821621 30229 raft_consensus.cc:348] T 4cea6495041642f58e1eb5d26fc10df5 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:53.825286 30229 raft_consensus.cc:374] T 4cea6495041642f58e1eb5d26fc10df5 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:53.825449 30229 raft_consensus.cc:729] T 4cea6495041642f58e1eb5d26fc10df5 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:14:53.825803 30229 consensus_queue.cc:260] T 4cea6495041642f58e1eb5d26fc10df5 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:53.826829 30232 ts_tablet_manager.cc:1875] T 1a3e938ab64e4fc09462c2dbfcee015e P 1cb67158133e4648a6eceba32d4dbba3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:53.827176 30231 tablet_bootstrap.cc:654] T f94a6ca24bf748c1ac4f12df761d537d P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:53.832118 30237 raft_consensus.cc:374] T 2d724f07a884486f9c5298ff585d62cb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:53.832751 30229 ts_tablet_manager.cc:1398] T 4cea6495041642f58e1eb5d26fc10df5 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20240502 14:14:53.832926 30237 raft_consensus.cc:729] T 2d724f07a884486f9c5298ff585d62cb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:53.833084 30233 tablet_bootstrap.cc:654] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:53.833303 30237 consensus_queue.cc:260] T 2d724f07a884486f9c5298ff585d62cb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:53.834616 30229 tablet_bootstrap.cc:492] T f94a6ca24bf748c1ac4f12df761d537d P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:14:53.835242 30235 tablet_bootstrap.cc:654] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:53.837297 30237 ts_tablet_manager.cc:1398] T 2d724f07a884486f9c5298ff585d62cb P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20240502 14:14:53.839149 30231 tablet_bootstrap.cc:492] T f94a6ca24bf748c1ac4f12df761d537d P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:14:53.840135 30229 tablet_bootstrap.cc:654] T f94a6ca24bf748c1ac4f12df761d537d P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:53.842569 30231 ts_tablet_manager.cc:1367] T f94a6ca24bf748c1ac4f12df761d537d P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.000s
I20240502 14:14:53.843773 30231 raft_consensus.cc:348] T f94a6ca24bf748c1ac4f12df761d537d P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:53.844113 30231 raft_consensus.cc:374] T f94a6ca24bf748c1ac4f12df761d537d P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:53.844276 30231 raft_consensus.cc:729] T f94a6ca24bf748c1ac4f12df761d537d P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:14:53.844792 30231 consensus_queue.cc:260] T f94a6ca24bf748c1ac4f12df761d537d P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:53.845474 30234 ts_tablet_manager.cc:1888] T 1a3e938ab64e4fc09462c2dbfcee015e P c1100e16b3b9495ca61d4f95e89b6a89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:53.845656 30234 log.cc:1196] T 1a3e938ab64e4fc09462c2dbfcee015e P c1100e16b3b9495ca61d4f95e89b6a89: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal/wals/1a3e938ab64e4fc09462c2dbfcee015e
I20240502 14:14:53.845695 30235 tablet_bootstrap.cc:492] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:53.845943 30235 ts_tablet_manager.cc:1367] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.004s
I20240502 14:14:53.846189 30234 ts_tablet_manager.cc:1909] T 1a3e938ab64e4fc09462c2dbfcee015e P c1100e16b3b9495ca61d4f95e89b6a89: Deleting consensus metadata
I20240502 14:14:53.846868 30235 raft_consensus.cc:348] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:53.847277 30235 raft_consensus.cc:374] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:53.847414 26271 catalog_manager.cc:4830] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): tablet 1a3e938ab64e4fc09462c2dbfcee015e (table rebalancer_dynamic_table_new_name [id=58dd440e1ee2463cb0df003412402dcd]) successfully deleted
I20240502 14:14:53.847486 30235 raft_consensus.cc:729] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:53.847787 30235 consensus_queue.cc:260] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:53.848644 30231 ts_tablet_manager.cc:1398] T f94a6ca24bf748c1ac4f12df761d537d P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:14:53.851231 30235 ts_tablet_manager.cc:1398] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 14:14:53.852421 30231 tablet_bootstrap.cc:492] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:14:53.854615 30231 tablet_bootstrap.cc:654] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:53.866396 30236 ts_tablet_manager.cc:1888] T 1a3e938ab64e4fc09462c2dbfcee015e P 4092902b3e674857a0147f1b4f2cdb86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:53.866824 30236 log.cc:1196] T 1a3e938ab64e4fc09462c2dbfcee015e P 4092902b3e674857a0147f1b4f2cdb86: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/wal/wals/1a3e938ab64e4fc09462c2dbfcee015e
I20240502 14:14:53.867980 30233 tablet_bootstrap.cc:492] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:14:53.868216 30233 ts_tablet_manager.cc:1367] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.054s user 0.003s sys 0.003s
I20240502 14:14:53.869280 30236 ts_tablet_manager.cc:1909] T 1a3e938ab64e4fc09462c2dbfcee015e P 4092902b3e674857a0147f1b4f2cdb86: Deleting consensus metadata
I20240502 14:14:53.869253 30233 raft_consensus.cc:348] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:53.869580 30233 raft_consensus.cc:374] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:53.869676 30233 raft_consensus.cc:729] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:14:53.869936 30233 consensus_queue.cc:260] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:53.870803 30233 ts_tablet_manager.cc:1398] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:14:53.875574 30233 tablet_bootstrap.cc:492] T 2d724f07a884486f9c5298ff585d62cb P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:14:53.875551 26283 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:14:53.877805 26272 catalog_manager.cc:4830] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097): tablet 1a3e938ab64e4fc09462c2dbfcee015e (table rebalancer_dynamic_table_new_name [id=58dd440e1ee2463cb0df003412402dcd]) successfully deleted
I20240502 14:14:53.877866 30229 tablet_bootstrap.cc:492] T f94a6ca24bf748c1ac4f12df761d537d P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:14:53.878064 30229 ts_tablet_manager.cc:1367] T f94a6ca24bf748c1ac4f12df761d537d P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.044s user 0.015s sys 0.013s
I20240502 14:14:53.879045 30229 raft_consensus.cc:348] T f94a6ca24bf748c1ac4f12df761d537d P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:53.879312 30229 raft_consensus.cc:374] T f94a6ca24bf748c1ac4f12df761d537d P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:53.879441 30229 raft_consensus.cc:729] T f94a6ca24bf748c1ac4f12df761d537d P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:14:53.879822 30229 consensus_queue.cc:260] T f94a6ca24bf748c1ac4f12df761d537d P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:53.880267 30233 tablet_bootstrap.cc:654] T 2d724f07a884486f9c5298ff585d62cb P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:53.877964 30232 ts_tablet_manager.cc:1888] T 1a3e938ab64e4fc09462c2dbfcee015e P 1cb67158133e4648a6eceba32d4dbba3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:53.884191 30232 log.cc:1196] T 1a3e938ab64e4fc09462c2dbfcee015e P 1cb67158133e4648a6eceba32d4dbba3: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal/wals/1a3e938ab64e4fc09462c2dbfcee015e
I20240502 14:14:53.884243 26819 consensus_queue.cc:237] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:53.884918 30232 ts_tablet_manager.cc:1909] T 1a3e938ab64e4fc09462c2dbfcee015e P 1cb67158133e4648a6eceba32d4dbba3: Deleting consensus metadata
I20240502 14:14:53.885524 30209 tablet_bootstrap.cc:492] T 825ae3e90e214a6f98394765edf6ff2d P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:53.886099 30209 tablet_bootstrap.cc:492] T 825ae3e90e214a6f98394765edf6ff2d P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap complete.
I20240502 14:14:53.886261 26271 catalog_manager.cc:4830] TS 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): tablet 1a3e938ab64e4fc09462c2dbfcee015e (table rebalancer_dynamic_table_new_name [id=58dd440e1ee2463cb0df003412402dcd]) successfully deleted
I20240502 14:14:53.889381 30231 tablet_bootstrap.cc:492] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:14:53.889900 30209 ts_tablet_manager.cc:1367] T 825ae3e90e214a6f98394765edf6ff2d P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.161s user 0.082s sys 0.030s
I20240502 14:14:53.891111 30209 raft_consensus.cc:348] T 825ae3e90e214a6f98394765edf6ff2d P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:53.891404 30209 raft_consensus.cc:729] T 825ae3e90e214a6f98394765edf6ff2d P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:14:53.891671 30209 consensus_queue.cc:260] T 825ae3e90e214a6f98394765edf6ff2d P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:53.892326 30209 ts_tablet_manager.cc:1398] T 825ae3e90e214a6f98394765edf6ff2d P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:14:53.893954 26578 tablet_copy_service.cc:342] P ba187e38a2754d60a12de9ac4856430a: Request end of tablet copy session 4092902b3e674857a0147f1b4f2cdb86-825ae3e90e214a6f98394765edf6ff2d received from {username='slave'} at 127.19.94.200:52883
I20240502 14:14:53.894181 26578 tablet_copy_service.cc:433] P ba187e38a2754d60a12de9ac4856430a: ending tablet copy session 4092902b3e674857a0147f1b4f2cdb86-825ae3e90e214a6f98394765edf6ff2d on tablet 825ae3e90e214a6f98394765edf6ff2d with peer 4092902b3e674857a0147f1b4f2cdb86
I20240502 14:14:53.895042 30229 ts_tablet_manager.cc:1398] T f94a6ca24bf748c1ac4f12df761d537d P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 14:14:53.895152 26537 tablet_service.cc:1467] Processing CreateTablet for tablet d4e21aeef6f0412fb7213a676fa24474 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dcbe9732b62c4a90a047e3270533663b]), partition=
I20240502 14:14:53.895381 29154 tablet_service.cc:1467] Processing CreateTablet for tablet d4e21aeef6f0412fb7213a676fa24474 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dcbe9732b62c4a90a047e3270533663b]), partition=
I20240502 14:14:53.895593 26537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4e21aeef6f0412fb7213a676fa24474. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:53.895921 29154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4e21aeef6f0412fb7213a676fa24474. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:53.901028 29423 tablet_service.cc:1467] Processing CreateTablet for tablet d4e21aeef6f0412fb7213a676fa24474 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dcbe9732b62c4a90a047e3270533663b]), partition=
I20240502 14:14:53.905388 29423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4e21aeef6f0412fb7213a676fa24474. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:53.908655 26423 raft_consensus.cc:1261] T 6b4745b1153b4acd999ac3a5c137ff53 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:53.909121 26557 raft_consensus.cc:1261] T 6b4745b1153b4acd999ac3a5c137ff53 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:53.910086 29226 consensus_queue.cc:1035] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:53.910492 30231 ts_tablet_manager.cc:1367] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.058s user 0.000s sys 0.008s
I20240502 14:14:53.910449 29791 consensus_queue.cc:1035] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:53.915524 30231 raft_consensus.cc:348] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:53.916030 30231 raft_consensus.cc:374] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:53.916164 30231 raft_consensus.cc:729] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:14:53.916455 30239 tablet_bootstrap.cc:492] T 2d724f07a884486f9c5298ff585d62cb P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:14:53.916476 30231 consensus_queue.cc:260] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:53.918830 30231 ts_tablet_manager.cc:1398] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 14:14:53.919575 30231 tablet_bootstrap.cc:492] T d4e21aeef6f0412fb7213a676fa24474 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:14:53.919750 29226 raft_consensus.cc:2932] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:14:53.920956 30239 ts_tablet_manager.cc:1367] T 2d724f07a884486f9c5298ff585d62cb P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.108s user 0.006s sys 0.009s
I20240502 14:14:53.921200 30231 tablet_bootstrap.cc:654] T d4e21aeef6f0412fb7213a676fa24474 P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:53.922204 30239 raft_consensus.cc:348] T 2d724f07a884486f9c5298ff585d62cb P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:53.922550 30239 raft_consensus.cc:374] T 2d724f07a884486f9c5298ff585d62cb P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:53.922689 30239 raft_consensus.cc:729] T 2d724f07a884486f9c5298ff585d62cb P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:14:53.922964 30239 consensus_queue.cc:260] T 2d724f07a884486f9c5298ff585d62cb P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:53.925184 30239 ts_tablet_manager.cc:1398] T 2d724f07a884486f9c5298ff585d62cb P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:14:53.925370 30231 tablet_bootstrap.cc:492] T d4e21aeef6f0412fb7213a676fa24474 P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:14:53.925522 30231 ts_tablet_manager.cc:1367] T d4e21aeef6f0412fb7213a676fa24474 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240502 14:14:53.926268 30231 raft_consensus.cc:348] T d4e21aeef6f0412fb7213a676fa24474 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:53.926435 30231 raft_consensus.cc:374] T d4e21aeef6f0412fb7213a676fa24474 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:53.926528 30231 raft_consensus.cc:729] T d4e21aeef6f0412fb7213a676fa24474 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:14:53.926748 30231 consensus_queue.cc:260] T d4e21aeef6f0412fb7213a676fa24474 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:53.927708 30233 tablet_bootstrap.cc:492] T 2d724f07a884486f9c5298ff585d62cb P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:14:53.927888 30233 ts_tablet_manager.cc:1367] T 2d724f07a884486f9c5298ff585d62cb P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.052s user 0.006s sys 0.004s
I20240502 14:14:53.928687 30233 raft_consensus.cc:348] T 2d724f07a884486f9c5298ff585d62cb P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:53.928920 30233 raft_consensus.cc:374] T 2d724f07a884486f9c5298ff585d62cb P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:53.929003 30233 raft_consensus.cc:729] T 2d724f07a884486f9c5298ff585d62cb P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:14:53.929256 30233 consensus_queue.cc:260] T 2d724f07a884486f9c5298ff585d62cb P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:53.933607 26423 raft_consensus.cc:2932] T 6b4745b1153b4acd999ac3a5c137ff53 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:14:53.935158 30233 ts_tablet_manager.cc:1398] T 2d724f07a884486f9c5298ff585d62cb P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:14:53.935669 30231 ts_tablet_manager.cc:1398] T d4e21aeef6f0412fb7213a676fa24474 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.010s user 0.001s sys 0.002s
I20240502 14:14:53.937090 30233 tablet_bootstrap.cc:492] T d4e21aeef6f0412fb7213a676fa24474 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:14:53.938089 26557 raft_consensus.cc:2932] T 6b4745b1153b4acd999ac3a5c137ff53 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:14:53.939116 30233 tablet_bootstrap.cc:654] T d4e21aeef6f0412fb7213a676fa24474 P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:53.941115 30258 tablet_bootstrap.cc:492] T d4e21aeef6f0412fb7213a676fa24474 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:53.941637 26283 catalog_manager.cc:5484] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: config changed from index -1 to 4, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:14:53.946033 29918 raft_consensus.cc:482] T 2d724f07a884486f9c5298ff585d62cb P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:53.946216 29918 raft_consensus.cc:504] T 2d724f07a884486f9c5298ff585d62cb P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:53.946759 30258 tablet_bootstrap.cc:654] T d4e21aeef6f0412fb7213a676fa24474 P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:53.947172 29918 leader_election.cc:288] T 2d724f07a884486f9c5298ff585d62cb P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
W20240502 14:14:53.949103 26735 tablet.cc:2369] T 2d724f07a884486f9c5298ff585d62cb P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:14:53.961452 30258 tablet_bootstrap.cc:492] T d4e21aeef6f0412fb7213a676fa24474 P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:53.961799 30258 ts_tablet_manager.cc:1367] T d4e21aeef6f0412fb7213a676fa24474 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.007s
W20240502 14:14:53.963035 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b4745b1153b4acd999ac3a5c137ff53. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:53.962985 30258 raft_consensus.cc:348] T d4e21aeef6f0412fb7213a676fa24474 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:53.963510 30258 raft_consensus.cc:374] T d4e21aeef6f0412fb7213a676fa24474 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:53.963685 30258 raft_consensus.cc:729] T d4e21aeef6f0412fb7213a676fa24474 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:53.964043 30258 consensus_queue.cc:260] T d4e21aeef6f0412fb7213a676fa24474 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:53.966341 30258 ts_tablet_manager.cc:1398] T d4e21aeef6f0412fb7213a676fa24474 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20240502 14:14:53.969563 30233 tablet_bootstrap.cc:492] T d4e21aeef6f0412fb7213a676fa24474 P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:14:53.969808 30233 ts_tablet_manager.cc:1367] T d4e21aeef6f0412fb7213a676fa24474 P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.000s
I20240502 14:14:53.970947 30233 raft_consensus.cc:348] T d4e21aeef6f0412fb7213a676fa24474 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:53.971212 30233 raft_consensus.cc:374] T d4e21aeef6f0412fb7213a676fa24474 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:53.971336 30233 raft_consensus.cc:729] T d4e21aeef6f0412fb7213a676fa24474 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:14:53.971581 30233 consensus_queue.cc:260] T d4e21aeef6f0412fb7213a676fa24474 P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:53.972319 30233 ts_tablet_manager.cc:1398] T d4e21aeef6f0412fb7213a676fa24474 P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:53.973857 30008 raft_consensus.cc:482] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:53.974056 30008 raft_consensus.cc:504] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:53.974777 30008 leader_election.cc:288] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:14:53.975569 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cea6495041642f58e1eb5d26fc10df5" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:14:53.975718 26423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cea6495041642f58e1eb5d26fc10df5" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:14:53.975837 29309 raft_consensus.cc:2454] T 4cea6495041642f58e1eb5d26fc10df5 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 0.
I20240502 14:14:53.975977 26423 raft_consensus.cc:2454] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 0.
I20240502 14:14:53.976917 29378 leader_election.cc:302] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:14:53.977366 30008 raft_consensus.cc:2781] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:53.977485 30008 raft_consensus.cc:482] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:53.977592 30008 raft_consensus.cc:3037] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:53.982100 30008 raft_consensus.cc:504] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:53.984022 30008 leader_election.cc:288] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 election: Requested vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:14:53.984624 26423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cea6495041642f58e1eb5d26fc10df5" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:14:53.985016 26423 raft_consensus.cc:3037] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:53.992257 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cea6495041642f58e1eb5d26fc10df5" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
I20240502 14:14:53.992684 29309 raft_consensus.cc:3037] T 4cea6495041642f58e1eb5d26fc10df5 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:53.997119 29309 raft_consensus.cc:2454] T 4cea6495041642f58e1eb5d26fc10df5 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 1.
I20240502 14:14:53.997119 26423 raft_consensus.cc:2454] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 1.
I20240502 14:14:53.997714 29378 leader_election.cc:302] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:14:53.998118 30008 raft_consensus.cc:2781] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:54.005152 29575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d724f07a884486f9c5298ff585d62cb" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86" is_pre_election: true
I20240502 14:14:54.005453 29575 raft_consensus.cc:2454] T 2d724f07a884486f9c5298ff585d62cb P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 0.
I20240502 14:14:54.008322 29112 leader_election.cc:302] T 2d724f07a884486f9c5298ff585d62cb P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 4092902b3e674857a0147f1b4f2cdb86; no voters:
I20240502 14:14:54.010512 29918 raft_consensus.cc:2781] T 2d724f07a884486f9c5298ff585d62cb P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:54.010700 29918 raft_consensus.cc:482] T 2d724f07a884486f9c5298ff585d62cb P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:54.010805 29918 raft_consensus.cc:3037] T 2d724f07a884486f9c5298ff585d62cb P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:54.014618 29918 raft_consensus.cc:504] T 2d724f07a884486f9c5298ff585d62cb P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:54.015421 29918 leader_election.cc:288] T 2d724f07a884486f9c5298ff585d62cb P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 election: Requested vote from peers 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:14:54.016413 29575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d724f07a884486f9c5298ff585d62cb" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86"
I20240502 14:14:54.016680 29575 raft_consensus.cc:3037] T 2d724f07a884486f9c5298ff585d62cb P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:54.021677 29575 raft_consensus.cc:2454] T 2d724f07a884486f9c5298ff585d62cb P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 1.
I20240502 14:14:54.022979 29112 leader_election.cc:302] T 2d724f07a884486f9c5298ff585d62cb P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 4092902b3e674857a0147f1b4f2cdb86; no voters:
I20240502 14:14:54.023290 29918 raft_consensus.cc:2781] T 2d724f07a884486f9c5298ff585d62cb P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:54.023733 29918 raft_consensus.cc:686] T 2d724f07a884486f9c5298ff585d62cb P 1cb67158133e4648a6eceba32d4dbba3 [term 1 LEADER]: Becoming Leader. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Running, Role: LEADER
I20240502 14:14:54.024085 29918 consensus_queue.cc:237] T 2d724f07a884486f9c5298ff585d62cb P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:54.025585 29645 raft_consensus.cc:482] T d4e21aeef6f0412fb7213a676fa24474 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:54.025768 29645 raft_consensus.cc:504] T d4e21aeef6f0412fb7213a676fa24474 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:54.026598 29645 leader_election.cc:288] T d4e21aeef6f0412fb7213a676fa24474 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:14:54.028750 26283 catalog_manager.cc:5484] T 2d724f07a884486f9c5298ff585d62cb P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197). New cstate: current_term: 1 leader_uuid: "1cb67158133e4648a6eceba32d4dbba3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:14:54.051110 29918 raft_consensus.cc:482] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:54.051443 26689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d724f07a884486f9c5298ff585d62cb" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:14:54.051368 29918 raft_consensus.cc:504] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:54.051674 26689 raft_consensus.cc:2454] T 2d724f07a884486f9c5298ff585d62cb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 0.
I20240502 14:14:54.052058 26687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d724f07a884486f9c5298ff585d62cb" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:14:54.052244 26687 raft_consensus.cc:3037] T 2d724f07a884486f9c5298ff585d62cb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:54.052374 29918 leader_election.cc:288] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:14:54.055263 26687 raft_consensus.cc:2454] T 2d724f07a884486f9c5298ff585d62cb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 1.
I20240502 14:14:54.064213 29575 raft_consensus.cc:1203] T 825ae3e90e214a6f98394765edf6ff2d P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:54.076108 30279 ts_tablet_manager.cc:902] T 981e25d29abc44299dc115d6df835c58 P 1cb67158133e4648a6eceba32d4dbba3: Initiating tablet copy from peer e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:54.081192 30008 raft_consensus.cc:686] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 LEADER]: Becoming Leader. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Running, Role: LEADER
I20240502 14:14:54.081704 30008 consensus_queue.cc:237] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:54.083623 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f94a6ca24bf748c1ac4f12df761d537d" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:14:54.083894 29309 raft_consensus.cc:2454] T f94a6ca24bf748c1ac4f12df761d537d P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 0.
I20240502 14:14:54.084524 29111 leader_election.cc:302] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:14:54.085095 29918 raft_consensus.cc:2781] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:54.085199 29918 raft_consensus.cc:482] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:54.085305 29918 raft_consensus.cc:3037] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:54.088506 29918 raft_consensus.cc:504] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:54.089469 29918 leader_election.cc:288] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 election: Requested vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:14:54.091382 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f94a6ca24bf748c1ac4f12df761d537d" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
I20240502 14:14:54.091619 29309 raft_consensus.cc:3037] T f94a6ca24bf748c1ac4f12df761d537d P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:54.094203 29309 raft_consensus.cc:2454] T f94a6ca24bf748c1ac4f12df761d537d P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 1.
I20240502 14:14:54.096320 29176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4e21aeef6f0412fb7213a676fa24474" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:14:54.096585 29176 raft_consensus.cc:2454] T d4e21aeef6f0412fb7213a676fa24474 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 0.
I20240502 14:14:54.098768 29442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f94a6ca24bf748c1ac4f12df761d537d" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:14:54.098902 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f94a6ca24bf748c1ac4f12df761d537d" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
I20240502 14:14:54.099010 29111 leader_election.cc:302] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:14:54.099045 29442 raft_consensus.cc:2454] T f94a6ca24bf748c1ac4f12df761d537d P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 0.
I20240502 14:14:54.099431 26493 leader_election.cc:302] T d4e21aeef6f0412fb7213a676fa24474 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:54.100279 29645 raft_consensus.cc:2781] T d4e21aeef6f0412fb7213a676fa24474 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:54.100426 29645 raft_consensus.cc:482] T d4e21aeef6f0412fb7213a676fa24474 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:54.100533 29645 raft_consensus.cc:3037] T d4e21aeef6f0412fb7213a676fa24474 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:54.101716 29918 raft_consensus.cc:2781] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:54.102447 29918 raft_consensus.cc:686] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3 [term 1 LEADER]: Becoming Leader. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Running, Role: LEADER
I20240502 14:14:54.102734 29918 consensus_queue.cc:237] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:54.103646 29645 raft_consensus.cc:504] T d4e21aeef6f0412fb7213a676fa24474 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:54.105430 29176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4e21aeef6f0412fb7213a676fa24474" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:14:54.105649 29176 raft_consensus.cc:3037] T d4e21aeef6f0412fb7213a676fa24474 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:54.104372 29645 leader_election.cc:288] T d4e21aeef6f0412fb7213a676fa24474 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Requested vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:14:54.108451 29176 raft_consensus.cc:2454] T d4e21aeef6f0412fb7213a676fa24474 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:14:54.108919 26493 leader_election.cc:302] T d4e21aeef6f0412fb7213a676fa24474 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:54.109493 29645 raft_consensus.cc:2781] T d4e21aeef6f0412fb7213a676fa24474 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:54.110721 29645 raft_consensus.cc:686] T d4e21aeef6f0412fb7213a676fa24474 P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Becoming Leader. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Running, Role: LEADER
I20240502 14:14:54.113020 29645 consensus_queue.cc:237] T d4e21aeef6f0412fb7213a676fa24474 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:54.120702 26283 catalog_manager.cc:5484] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199). New cstate: current_term: 1 leader_uuid: "68a3c2b17ed94e06aba66ed4178132ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:14:54.128508 29442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4e21aeef6f0412fb7213a676fa24474" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:14:54.128810 29442 raft_consensus.cc:2454] T d4e21aeef6f0412fb7213a676fa24474 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 0.
I20240502 14:14:54.129832 29442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4e21aeef6f0412fb7213a676fa24474" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
I20240502 14:14:54.130043 29442 raft_consensus.cc:3037] T d4e21aeef6f0412fb7213a676fa24474 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:54.133205 29442 raft_consensus.cc:2454] T d4e21aeef6f0412fb7213a676fa24474 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:14:54.133461 26283 catalog_manager.cc:5484] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197). New cstate: current_term: 1 leader_uuid: "1cb67158133e4648a6eceba32d4dbba3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:14:54.148078 26285 catalog_manager.cc:5484] T d4e21aeef6f0412fb7213a676fa24474 P ba187e38a2754d60a12de9ac4856430a reported cstate change: term changed from 0 to 1, leader changed from <none> to ba187e38a2754d60a12de9ac4856430a (127.19.94.194). New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:14:54.150719 30279 tablet_copy_client.cc:308] T 981e25d29abc44299dc115d6df835c58 P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.196:32843
I20240502 14:14:54.151997 26839 tablet_copy_service.cc:140] P e4cfb8fc5a474177af6848a6efa13a27: Received BeginTabletCopySession request for tablet 981e25d29abc44299dc115d6df835c58 from peer 1cb67158133e4648a6eceba32d4dbba3 ({username='slave'} at 127.19.94.197:46803)
I20240502 14:14:54.152227 26839 tablet_copy_service.cc:161] P e4cfb8fc5a474177af6848a6efa13a27: Beginning new tablet copy session on tablet 981e25d29abc44299dc115d6df835c58 from peer 1cb67158133e4648a6eceba32d4dbba3 at {username='slave'} at 127.19.94.197:46803: session id = 1cb67158133e4648a6eceba32d4dbba3-981e25d29abc44299dc115d6df835c58
I20240502 14:14:54.156019 26839 tablet_copy_source_session.cc:205] T 981e25d29abc44299dc115d6df835c58 P e4cfb8fc5a474177af6848a6efa13a27: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:54.161322 30279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 981e25d29abc44299dc115d6df835c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:54.171742 30279 tablet_copy_client.cc:783] T 981e25d29abc44299dc115d6df835c58 P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:54.172048 30279 tablet_copy_client.cc:647] T 981e25d29abc44299dc115d6df835c58 P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:54.174314 30279 tablet_copy_client.cc:523] T 981e25d29abc44299dc115d6df835c58 P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:54.177752 30279 tablet_bootstrap.cc:492] T 981e25d29abc44299dc115d6df835c58 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:14:54.243683 30279 tablet_bootstrap.cc:492] T 981e25d29abc44299dc115d6df835c58 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:54.244347 30279 tablet_bootstrap.cc:492] T 981e25d29abc44299dc115d6df835c58 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap complete.
I20240502 14:14:54.244899 30279 ts_tablet_manager.cc:1367] T 981e25d29abc44299dc115d6df835c58 P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.067s user 0.046s sys 0.003s
I20240502 14:14:54.246018 30279 raft_consensus.cc:348] T 981e25d29abc44299dc115d6df835c58 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:54.246434 30279 raft_consensus.cc:729] T 981e25d29abc44299dc115d6df835c58 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:14:54.246757 30279 consensus_queue.cc:260] T 981e25d29abc44299dc115d6df835c58 P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:54.252790 30279 ts_tablet_manager.cc:1398] T 981e25d29abc44299dc115d6df835c58 P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:54.254650 26839 tablet_copy_service.cc:342] P e4cfb8fc5a474177af6848a6efa13a27: Request end of tablet copy session 1cb67158133e4648a6eceba32d4dbba3-981e25d29abc44299dc115d6df835c58 received from {username='slave'} at 127.19.94.197:46803
I20240502 14:14:54.254904 26839 tablet_copy_service.cc:433] P e4cfb8fc5a474177af6848a6efa13a27: ending tablet copy session 1cb67158133e4648a6eceba32d4dbba3-981e25d29abc44299dc115d6df835c58 on tablet 981e25d29abc44299dc115d6df835c58 with peer 1cb67158133e4648a6eceba32d4dbba3
I20240502 14:14:54.288523 30312 ts_tablet_manager.cc:902] T 6b4745b1153b4acd999ac3a5c137ff53 P c1100e16b3b9495ca61d4f95e89b6a89: Initiating tablet copy from peer e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:54.291558 26285 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:14:54.312321 30312 tablet_copy_client.cc:308] T 6b4745b1153b4acd999ac3a5c137ff53 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.196:32843
I20240502 14:14:54.323221 29176 raft_consensus.cc:1261] T d4e21aeef6f0412fb7213a676fa24474 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:54.324236 29718 consensus_queue.cc:1035] T d4e21aeef6f0412fb7213a676fa24474 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:54.332341 29442 raft_consensus.cc:1261] T d4e21aeef6f0412fb7213a676fa24474 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:54.333045 29645 consensus_queue.cc:1035] T d4e21aeef6f0412fb7213a676fa24474 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:54.352331 30323 tablet.cc:1722] T d4e21aeef6f0412fb7213a676fa24474 P ba187e38a2754d60a12de9ac4856430a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:54.353222 30323 tablet.cc:1620] T d4e21aeef6f0412fb7213a676fa24474 P ba187e38a2754d60a12de9ac4856430a: MemRowSet was empty: no flush needed.
I20240502 14:14:54.359922 30324 tablet.cc:1722] T d4e21aeef6f0412fb7213a676fa24474 P 1cb67158133e4648a6eceba32d4dbba3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:54.361339 26279 catalog_manager.cc:5576] TS 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715) does not have the latest schema for tablet d4e21aeef6f0412fb7213a676fa24474 (table rebalancer_dynamic_table [id=dcbe9732b62c4a90a047e3270533663b]). Expected version 1 got 0
I20240502 14:14:54.361461 30324 tablet.cc:1620] T d4e21aeef6f0412fb7213a676fa24474 P 1cb67158133e4648a6eceba32d4dbba3: MemRowSet was empty: no flush needed.
I20240502 14:14:54.368073 26279 catalog_manager.cc:5576] TS 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781) does not have the latest schema for tablet d4e21aeef6f0412fb7213a676fa24474 (table rebalancer_dynamic_table [id=dcbe9732b62c4a90a047e3270533663b]). Expected version 1 got 0
I20240502 14:14:54.369249 30327 tablet.cc:1722] T d4e21aeef6f0412fb7213a676fa24474 P 68a3c2b17ed94e06aba66ed4178132ab: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:54.370049 30327 tablet.cc:1620] T d4e21aeef6f0412fb7213a676fa24474 P 68a3c2b17ed94e06aba66ed4178132ab: MemRowSet was empty: no flush needed.
I20240502 14:14:54.370040 26285 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=dcbe9732b62c4a90a047e3270533663b] alter complete (version 1)
I20240502 14:14:54.392148 26839 tablet_copy_service.cc:140] P e4cfb8fc5a474177af6848a6efa13a27: Received BeginTabletCopySession request for tablet 6b4745b1153b4acd999ac3a5c137ff53 from peer c1100e16b3b9495ca61d4f95e89b6a89 ({username='slave'} at 127.19.94.198:48649)
I20240502 14:14:54.392364 26839 tablet_copy_service.cc:161] P e4cfb8fc5a474177af6848a6efa13a27: Beginning new tablet copy session on tablet 6b4745b1153b4acd999ac3a5c137ff53 from peer c1100e16b3b9495ca61d4f95e89b6a89 at {username='slave'} at 127.19.94.198:48649: session id = c1100e16b3b9495ca61d4f95e89b6a89-6b4745b1153b4acd999ac3a5c137ff53
I20240502 14:14:54.394882 26839 tablet_copy_source_session.cc:205] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:54.397006 30312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b4745b1153b4acd999ac3a5c137ff53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:54.402406 30312 tablet_copy_client.cc:783] T 6b4745b1153b4acd999ac3a5c137ff53 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:54.402844 30312 tablet_copy_client.cc:647] T 6b4745b1153b4acd999ac3a5c137ff53 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:54.405354 30312 tablet_copy_client.cc:523] T 6b4745b1153b4acd999ac3a5c137ff53 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:54.411326 30312 tablet_bootstrap.cc:492] T 6b4745b1153b4acd999ac3a5c137ff53 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:14:54.427592 29154 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:14:54.441186 29553 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:14:54.476084 29423 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:14:54.476747 26668 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:14:54.477269 26276 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:14:54.479032 29287 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:54.489082 26797 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:14:54.496146 30323 tablet.cc:1722] T d4e21aeef6f0412fb7213a676fa24474 P ba187e38a2754d60a12de9ac4856430a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:54.496860 30323 tablet.cc:1620] T d4e21aeef6f0412fb7213a676fa24474 P ba187e38a2754d60a12de9ac4856430a: MemRowSet was empty: no flush needed.
I20240502 14:14:54.498243 30008 consensus_queue.cc:1035] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:54.518224 30008 consensus_queue.cc:1035] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:54.527868 29918 consensus_queue.cc:1035] T 2d724f07a884486f9c5298ff585d62cb P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:54.528308 26276 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=dcbe9732b62c4a90a047e3270533663b] alter complete (version 2)
I20240502 14:14:54.532124 30327 tablet.cc:1722] T d4e21aeef6f0412fb7213a676fa24474 P 68a3c2b17ed94e06aba66ed4178132ab: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:54.532732 30327 tablet.cc:1620] T d4e21aeef6f0412fb7213a676fa24474 P 68a3c2b17ed94e06aba66ed4178132ab: MemRowSet was empty: no flush needed.
I20240502 14:14:54.544013 29442 raft_consensus.cc:3037] T f94a6ca24bf748c1ac4f12df761d537d P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:54.552942 29174 raft_consensus.cc:1203] T 981e25d29abc44299dc115d6df835c58 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:54.556983 29918 consensus_queue.cc:1035] T 2d724f07a884486f9c5298ff585d62cb P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:54.573103 30324 tablet.cc:1722] T d4e21aeef6f0412fb7213a676fa24474 P 1cb67158133e4648a6eceba32d4dbba3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:54.573644 30324 tablet.cc:1620] T d4e21aeef6f0412fb7213a676fa24474 P 1cb67158133e4648a6eceba32d4dbba3: MemRowSet was empty: no flush needed.
I20240502 14:14:54.574251 30274 consensus_queue.cc:1035] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:54.585877 26406 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:14:54.598332 26537 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:14:54.601619 29918 consensus_queue.cc:1035] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:54.638685 26276 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:14:54.649636 30323 tablet.cc:1722] T d4e21aeef6f0412fb7213a676fa24474 P ba187e38a2754d60a12de9ac4856430a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:54.656087 30324 tablet.cc:1722] T d4e21aeef6f0412fb7213a676fa24474 P 1cb67158133e4648a6eceba32d4dbba3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:54.665172 26279 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table_new_name [id=dcbe9732b62c4a90a047e3270533663b] alter complete (version 3)
I20240502 14:14:54.667305 30327 tablet.cc:1722] T d4e21aeef6f0412fb7213a676fa24474 P 68a3c2b17ed94e06aba66ed4178132ab: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:54.669237 26279 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_extra_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:54.689275 26537 tablet_service.cc:1467] Processing CreateTablet for tablet a68c11570ad84193b88eede6c26614e3 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=48a10accbab847d6b456b86bacb19a13]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:54.689278 29154 tablet_service.cc:1467] Processing CreateTablet for tablet a68c11570ad84193b88eede6c26614e3 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=48a10accbab847d6b456b86bacb19a13]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:54.689774 26537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a68c11570ad84193b88eede6c26614e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:54.689802 29154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a68c11570ad84193b88eede6c26614e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:54.695225 26406 tablet_service.cc:1467] Processing CreateTablet for tablet a68c11570ad84193b88eede6c26614e3 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=48a10accbab847d6b456b86bacb19a13]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:54.695724 26406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a68c11570ad84193b88eede6c26614e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:54.701670 26668 tablet_service.cc:1467] Processing CreateTablet for tablet d0838f171f3146a181ff1f3e843cdec1 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=48a10accbab847d6b456b86bacb19a13]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:54.702232 26668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0838f171f3146a181ff1f3e843cdec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:54.702553 29287 tablet_service.cc:1467] Processing CreateTablet for tablet d0838f171f3146a181ff1f3e843cdec1 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=48a10accbab847d6b456b86bacb19a13]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:54.703069 29287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0838f171f3146a181ff1f3e843cdec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:54.703133 29553 tablet_service.cc:1467] Processing CreateTablet for tablet d0838f171f3146a181ff1f3e843cdec1 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=48a10accbab847d6b456b86bacb19a13]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:54.703542 29553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0838f171f3146a181ff1f3e843cdec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:54.706703 26797 tablet_service.cc:1467] Processing CreateTablet for tablet c32ea761f2834d758529890e5b496f88 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=48a10accbab847d6b456b86bacb19a13]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:54.707124 26797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c32ea761f2834d758529890e5b496f88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:54.710408 29156 tablet_service.cc:1467] Processing CreateTablet for tablet c32ea761f2834d758529890e5b496f88 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=48a10accbab847d6b456b86bacb19a13]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:54.710847 29156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c32ea761f2834d758529890e5b496f88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:54.721684 30354 tablet_bootstrap.cc:492] T a68c11570ad84193b88eede6c26614e3 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:14:54.724265 30354 tablet_bootstrap.cc:654] T a68c11570ad84193b88eede6c26614e3 P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:54.733253 30359 tablet_bootstrap.cc:492] T a68c11570ad84193b88eede6c26614e3 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:54.734351 29288 tablet_service.cc:1467] Processing CreateTablet for tablet c32ea761f2834d758529890e5b496f88 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=48a10accbab847d6b456b86bacb19a13]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:54.735687 29288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c32ea761f2834d758529890e5b496f88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:54.748664 30359 tablet_bootstrap.cc:654] T a68c11570ad84193b88eede6c26614e3 P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:54.752847 30354 tablet_bootstrap.cc:492] T a68c11570ad84193b88eede6c26614e3 P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:14:54.754137 30354 ts_tablet_manager.cc:1367] T a68c11570ad84193b88eede6c26614e3 P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.033s user 0.006s sys 0.000s
I20240502 14:14:54.757894 30354 raft_consensus.cc:348] T a68c11570ad84193b88eede6c26614e3 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:54.759479 30354 raft_consensus.cc:374] T a68c11570ad84193b88eede6c26614e3 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:54.759852 30359 tablet_bootstrap.cc:492] T a68c11570ad84193b88eede6c26614e3 P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:54.759860 30354 raft_consensus.cc:729] T a68c11570ad84193b88eede6c26614e3 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:14:54.760155 30359 ts_tablet_manager.cc:1367] T a68c11570ad84193b88eede6c26614e3 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.000s
I20240502 14:14:54.760407 30354 consensus_queue.cc:260] T a68c11570ad84193b88eede6c26614e3 P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:54.761324 30359 raft_consensus.cc:348] T a68c11570ad84193b88eede6c26614e3 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:54.761814 30359 raft_consensus.cc:374] T a68c11570ad84193b88eede6c26614e3 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:54.762027 30359 raft_consensus.cc:729] T a68c11570ad84193b88eede6c26614e3 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:54.762535 30359 consensus_queue.cc:260] T a68c11570ad84193b88eede6c26614e3 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:54.768498 30354 ts_tablet_manager.cc:1398] T a68c11570ad84193b88eede6c26614e3 P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240502 14:14:54.770792 26285 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:14:54.771026 26285 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:14:54.772802 30354 tablet_bootstrap.cc:492] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:14:54.775136 26285 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet d4e21aeef6f0412fb7213a676fa24474
I20240502 14:14:54.776227 29423 tablet_service.cc:1514] Processing DeleteTablet for tablet d4e21aeef6f0412fb7213a676fa24474 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:54 UTC) from {username='slave'} at 127.0.0.1:40652
I20240502 14:14:54.776525 29918 raft_consensus.cc:482] T a68c11570ad84193b88eede6c26614e3 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:54.776705 29918 raft_consensus.cc:504] T a68c11570ad84193b88eede6c26614e3 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:54.777137 26538 tablet_service.cc:1514] Processing DeleteTablet for tablet d4e21aeef6f0412fb7213a676fa24474 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:54 UTC) from {username='slave'} at 127.0.0.1:51696
I20240502 14:14:54.777451 29918 leader_election.cc:288] T a68c11570ad84193b88eede6c26614e3 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:14:54.777734 29154 tablet_service.cc:1514] Processing DeleteTablet for tablet d4e21aeef6f0412fb7213a676fa24474 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:54 UTC) from {username='slave'} at 127.0.0.1:55202
I20240502 14:14:54.778882 30359 ts_tablet_manager.cc:1398] T a68c11570ad84193b88eede6c26614e3 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.018s user 0.000s sys 0.002s
I20240502 14:14:54.778820 26423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a68c11570ad84193b88eede6c26614e3" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
W20240502 14:14:54.779631 29111 leader_election.cc:341] T a68c11570ad84193b88eede6c26614e3 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:54.778854 26557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a68c11570ad84193b88eede6c26614e3" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:54.780331 26557 raft_consensus.cc:2454] T a68c11570ad84193b88eede6c26614e3 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 0.
I20240502 14:14:54.781370 29111 leader_election.cc:302] T a68c11570ad84193b88eede6c26614e3 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, ba187e38a2754d60a12de9ac4856430a; no voters: 89af7f423cf44717a08d6b142f0ed49d
I20240502 14:14:54.782142 29918 raft_consensus.cc:2781] T a68c11570ad84193b88eede6c26614e3 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:54.782286 29918 raft_consensus.cc:482] T a68c11570ad84193b88eede6c26614e3 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:54.782397 29918 raft_consensus.cc:3037] T a68c11570ad84193b88eede6c26614e3 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:54.784613 30354 tablet_bootstrap.cc:654] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:54.803200 30362 tablet_bootstrap.cc:492] T d0838f171f3146a181ff1f3e843cdec1 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:14:54.806334 30362 tablet_bootstrap.cc:654] T d0838f171f3146a181ff1f3e843cdec1 P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:54.815475 30357 tablet_bootstrap.cc:492] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:54.817454 30357 tablet_bootstrap.cc:654] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:54.818982 30358 tablet_bootstrap.cc:492] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:54.820895 30358 tablet_bootstrap.cc:654] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:54.822670 30356 tablet_bootstrap.cc:492] T a68c11570ad84193b88eede6c26614e3 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:54.824378 30362 tablet_bootstrap.cc:492] T d0838f171f3146a181ff1f3e843cdec1 P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:14:54.824668 30362 ts_tablet_manager.cc:1367] T d0838f171f3146a181ff1f3e843cdec1 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.007s
I20240502 14:14:54.825049 30356 tablet_bootstrap.cc:654] T a68c11570ad84193b88eede6c26614e3 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:54.827216 30362 raft_consensus.cc:348] T d0838f171f3146a181ff1f3e843cdec1 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:54.828711 30362 raft_consensus.cc:374] T d0838f171f3146a181ff1f3e843cdec1 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:54.829094 30362 raft_consensus.cc:729] T d0838f171f3146a181ff1f3e843cdec1 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:14:54.829779 30362 consensus_queue.cc:260] T d0838f171f3146a181ff1f3e843cdec1 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:54.830026 30364 tablet_replica.cc:330] T d4e21aeef6f0412fb7213a676fa24474 P 68a3c2b17ed94e06aba66ed4178132ab: stopping tablet replica
I20240502 14:14:54.830288 30364 raft_consensus.cc:2229] T d4e21aeef6f0412fb7213a676fa24474 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:54.830586 30364 raft_consensus.cc:2258] T d4e21aeef6f0412fb7213a676fa24474 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:54.831804 30364 ts_tablet_manager.cc:1875] T d4e21aeef6f0412fb7213a676fa24474 P 68a3c2b17ed94e06aba66ed4178132ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:54.832927 30362 ts_tablet_manager.cc:1398] T d0838f171f3146a181ff1f3e843cdec1 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 14:14:54.833200 30362 tablet_bootstrap.cc:492] T c32ea761f2834d758529890e5b496f88 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:14:54.833804 30365 tablet_replica.cc:330] T d4e21aeef6f0412fb7213a676fa24474 P ba187e38a2754d60a12de9ac4856430a: stopping tablet replica
I20240502 14:14:54.834089 30365 raft_consensus.cc:2229] T d4e21aeef6f0412fb7213a676fa24474 P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:54.834545 30365 raft_consensus.cc:2258] T d4e21aeef6f0412fb7213a676fa24474 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:54.840915 30366 tablet_replica.cc:330] T d4e21aeef6f0412fb7213a676fa24474 P 1cb67158133e4648a6eceba32d4dbba3: stopping tablet replica
I20240502 14:14:54.841145 30366 raft_consensus.cc:2229] T d4e21aeef6f0412fb7213a676fa24474 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:54.841364 30366 raft_consensus.cc:2258] T d4e21aeef6f0412fb7213a676fa24474 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:54.842347 30362 tablet_bootstrap.cc:654] T c32ea761f2834d758529890e5b496f88 P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:54.842571 30366 ts_tablet_manager.cc:1875] T d4e21aeef6f0412fb7213a676fa24474 P 1cb67158133e4648a6eceba32d4dbba3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:54.840473 29918 raft_consensus.cc:504] T a68c11570ad84193b88eede6c26614e3 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:54.848870 30362 tablet_bootstrap.cc:492] T c32ea761f2834d758529890e5b496f88 P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:14:54.848930 26557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a68c11570ad84193b88eede6c26614e3" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:14:54.849069 30362 ts_tablet_manager.cc:1367] T c32ea761f2834d758529890e5b496f88 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20240502 14:14:54.849148 26557 raft_consensus.cc:3037] T a68c11570ad84193b88eede6c26614e3 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:54.849882 30362 raft_consensus.cc:348] T c32ea761f2834d758529890e5b496f88 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:54.851270 30362 raft_consensus.cc:374] T c32ea761f2834d758529890e5b496f88 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:54.851351 30362 raft_consensus.cc:729] T c32ea761f2834d758529890e5b496f88 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:14:54.851596 30362 consensus_queue.cc:260] T c32ea761f2834d758529890e5b496f88 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:54.851940 26557 raft_consensus.cc:2454] T a68c11570ad84193b88eede6c26614e3 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 1.
I20240502 14:14:54.852267 30362 ts_tablet_manager.cc:1398] T c32ea761f2834d758529890e5b496f88 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:54.852428 29111 leader_election.cc:302] T a68c11570ad84193b88eede6c26614e3 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:54.852783 30274 raft_consensus.cc:2781] T a68c11570ad84193b88eede6c26614e3 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:54.852844 29918 leader_election.cc:288] T a68c11570ad84193b88eede6c26614e3 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 election: Requested vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:14:54.859807 30274 raft_consensus.cc:686] T a68c11570ad84193b88eede6c26614e3 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 LEADER]: Becoming Leader. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Running, Role: LEADER
I20240502 14:14:54.860288 30366 ts_tablet_manager.cc:1888] T d4e21aeef6f0412fb7213a676fa24474 P 1cb67158133e4648a6eceba32d4dbba3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:54.860455 30366 log.cc:1196] T d4e21aeef6f0412fb7213a676fa24474 P 1cb67158133e4648a6eceba32d4dbba3: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal/wals/d4e21aeef6f0412fb7213a676fa24474
I20240502 14:14:54.860206 30274 consensus_queue.cc:237] T a68c11570ad84193b88eede6c26614e3 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:54.861088 30366 ts_tablet_manager.cc:1909] T d4e21aeef6f0412fb7213a676fa24474 P 1cb67158133e4648a6eceba32d4dbba3: Deleting consensus metadata
I20240502 14:14:54.862331 26271 catalog_manager.cc:4830] TS 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): tablet d4e21aeef6f0412fb7213a676fa24474 (table rebalancer_dynamic_table_new_name [id=dcbe9732b62c4a90a047e3270533663b]) successfully deleted
I20240502 14:14:54.862879 30365 ts_tablet_manager.cc:1875] T d4e21aeef6f0412fb7213a676fa24474 P ba187e38a2754d60a12de9ac4856430a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:54.854100 26423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a68c11570ad84193b88eede6c26614e3" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
W20240502 14:14:54.868754 29111 leader_election.cc:341] T a68c11570ad84193b88eede6c26614e3 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:54.871595 30354 tablet_bootstrap.cc:492] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:14:54.871860 30354 ts_tablet_manager.cc:1367] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.099s user 0.002s sys 0.003s
I20240502 14:14:54.872706 30354 raft_consensus.cc:348] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:54.872943 30354 raft_consensus.cc:374] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:54.873031 30354 raft_consensus.cc:729] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:14:54.873286 30354 consensus_queue.cc:260] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:54.873755 30354 ts_tablet_manager.cc:1398] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:14:54.876041 26279 catalog_manager.cc:5484] T a68c11570ad84193b88eede6c26614e3 P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197). New cstate: current_term: 1 leader_uuid: "1cb67158133e4648a6eceba32d4dbba3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:14:54.881621 30358 tablet_bootstrap.cc:492] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:54.881856 30358 ts_tablet_manager.cc:1367] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.063s user 0.003s sys 0.000s
I20240502 14:14:54.882023 30357 tablet_bootstrap.cc:492] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:54.882182 30357 ts_tablet_manager.cc:1367] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.067s user 0.000s sys 0.004s
I20240502 14:14:54.882790 30358 raft_consensus.cc:348] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:54.883153 30358 raft_consensus.cc:374] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:54.882882 26279 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:14:54.885769 30357 raft_consensus.cc:348] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:54.886082 30357 raft_consensus.cc:374] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:54.886322 30357 raft_consensus.cc:729] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:54.886636 30357 consensus_queue.cc:260] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:54.887308 30357 ts_tablet_manager.cc:1398] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 14:14:54.888473 30358 raft_consensus.cc:729] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:54.888808 30365 ts_tablet_manager.cc:1888] T d4e21aeef6f0412fb7213a676fa24474 P ba187e38a2754d60a12de9ac4856430a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:54.888991 30365 log.cc:1196] T d4e21aeef6f0412fb7213a676fa24474 P ba187e38a2754d60a12de9ac4856430a: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/d4e21aeef6f0412fb7213a676fa24474
I20240502 14:14:54.889526 30365 ts_tablet_manager.cc:1909] T d4e21aeef6f0412fb7213a676fa24474 P ba187e38a2754d60a12de9ac4856430a: Deleting consensus metadata
I20240502 14:14:54.891103 26271 catalog_manager.cc:4830] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): tablet d4e21aeef6f0412fb7213a676fa24474 (table rebalancer_dynamic_table_new_name [id=dcbe9732b62c4a90a047e3270533663b]) successfully deleted
I20240502 14:14:54.891403 30358 consensus_queue.cc:260] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:54.891994 30358 ts_tablet_manager.cc:1398] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240502 14:14:54.892593 30356 tablet_bootstrap.cc:492] T a68c11570ad84193b88eede6c26614e3 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:54.892755 30356 ts_tablet_manager.cc:1367] T a68c11570ad84193b88eede6c26614e3 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.070s user 0.005s sys 0.000s
I20240502 14:14:54.894734 30364 ts_tablet_manager.cc:1888] T d4e21aeef6f0412fb7213a676fa24474 P 68a3c2b17ed94e06aba66ed4178132ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:54.894884 30364 log.cc:1196] T d4e21aeef6f0412fb7213a676fa24474 P 68a3c2b17ed94e06aba66ed4178132ab: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal/wals/d4e21aeef6f0412fb7213a676fa24474
I20240502 14:14:54.895346 30364 ts_tablet_manager.cc:1909] T d4e21aeef6f0412fb7213a676fa24474 P 68a3c2b17ed94e06aba66ed4178132ab: Deleting consensus metadata
I20240502 14:14:54.896661 26272 catalog_manager.cc:4830] TS 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): tablet d4e21aeef6f0412fb7213a676fa24474 (table rebalancer_dynamic_table_new_name [id=dcbe9732b62c4a90a047e3270533663b]) successfully deleted
I20240502 14:14:54.896869 30356 raft_consensus.cc:348] T a68c11570ad84193b88eede6c26614e3 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:54.897071 30356 raft_consensus.cc:374] T a68c11570ad84193b88eede6c26614e3 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:54.897159 30356 raft_consensus.cc:729] T a68c11570ad84193b88eede6c26614e3 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:54.897408 30356 consensus_queue.cc:260] T a68c11570ad84193b88eede6c26614e3 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:54.898017 30356 ts_tablet_manager.cc:1398] T a68c11570ad84193b88eede6c26614e3 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 14:14:54.900918 30355 tablet_bootstrap.cc:492] T d0838f171f3146a181ff1f3e843cdec1 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:14:54.902937 30355 tablet_bootstrap.cc:654] T d0838f171f3146a181ff1f3e843cdec1 P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:54.920758 29423 tablet_service.cc:1467] Processing CreateTablet for tablet 48d9c7aad03d4b569c1932919d00e7d8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d6e507d0445241a9a4bdfa70480a61a7]), partition=
I20240502 14:14:54.920855 29553 tablet_service.cc:1467] Processing CreateTablet for tablet 48d9c7aad03d4b569c1932919d00e7d8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d6e507d0445241a9a4bdfa70480a61a7]), partition=
I20240502 14:14:54.921109 29288 tablet_service.cc:1467] Processing CreateTablet for tablet 48d9c7aad03d4b569c1932919d00e7d8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d6e507d0445241a9a4bdfa70480a61a7]), partition=
I20240502 14:14:54.921260 29423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48d9c7aad03d4b569c1932919d00e7d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:54.921275 29553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48d9c7aad03d4b569c1932919d00e7d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:54.921505 29288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48d9c7aad03d4b569c1932919d00e7d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:54.924597 30039 raft_consensus.cc:482] T a68c11570ad84193b88eede6c26614e3 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:54.924803 30039 raft_consensus.cc:504] T a68c11570ad84193b88eede6c26614e3 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:54.925689 30039 leader_election.cc:288] T a68c11570ad84193b88eede6c26614e3 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:14:54.926311 29174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a68c11570ad84193b88eede6c26614e3" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:14:54.926586 26557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a68c11570ad84193b88eede6c26614e3" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:54.926903 26557 raft_consensus.cc:2379] T a68c11570ad84193b88eede6c26614e3 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 89af7f423cf44717a08d6b142f0ed49d in current term 1: Already voted for candidate 1cb67158133e4648a6eceba32d4dbba3 in this term.
I20240502 14:14:54.927304 26362 leader_election.cc:302] T a68c11570ad84193b88eede6c26614e3 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d; no voters: 1cb67158133e4648a6eceba32d4dbba3, ba187e38a2754d60a12de9ac4856430a
I20240502 14:14:54.928273 30039 raft_consensus.cc:3037] T a68c11570ad84193b88eede6c26614e3 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:54.930801 30039 raft_consensus.cc:2726] T a68c11570ad84193b88eede6c26614e3 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:14:54.937084 30376 tablet_bootstrap.cc:492] T 48d9c7aad03d4b569c1932919d00e7d8 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:14:54.939313 30376 tablet_bootstrap.cc:654] T 48d9c7aad03d4b569c1932919d00e7d8 P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:54.941641 30362 tablet_bootstrap.cc:492] T 48d9c7aad03d4b569c1932919d00e7d8 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:14:54.945263 30362 tablet_bootstrap.cc:654] T 48d9c7aad03d4b569c1932919d00e7d8 P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:54.946496 30355 tablet_bootstrap.cc:492] T d0838f171f3146a181ff1f3e843cdec1 P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:14:54.946713 30355 ts_tablet_manager.cc:1367] T d0838f171f3146a181ff1f3e843cdec1 P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.046s user 0.008s sys 0.021s
I20240502 14:14:54.947659 30355 raft_consensus.cc:348] T d0838f171f3146a181ff1f3e843cdec1 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:54.947989 30355 raft_consensus.cc:374] T d0838f171f3146a181ff1f3e843cdec1 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:54.948129 30355 raft_consensus.cc:729] T d0838f171f3146a181ff1f3e843cdec1 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:14:54.948812 30355 consensus_queue.cc:260] T d0838f171f3146a181ff1f3e843cdec1 P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:54.950585 30355 ts_tablet_manager.cc:1398] T d0838f171f3146a181ff1f3e843cdec1 P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240502 14:14:54.950896 30355 tablet_bootstrap.cc:492] T 48d9c7aad03d4b569c1932919d00e7d8 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:14:54.953640 30274 raft_consensus.cc:482] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:54.953953 30355 tablet_bootstrap.cc:654] T 48d9c7aad03d4b569c1932919d00e7d8 P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:54.953858 30274 raft_consensus.cc:504] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:54.954814 30274 leader_election.cc:288] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:14:54.958086 29308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32ea761f2834d758529890e5b496f88" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:14:54.958446 29308 raft_consensus.cc:2454] T c32ea761f2834d758529890e5b496f88 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 0.
I20240502 14:14:54.959869 26819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32ea761f2834d758529890e5b496f88" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:14:54.960180 26819 raft_consensus.cc:2454] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 0.
I20240502 14:14:54.961556 29111 leader_election.cc:302] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:14:54.963881 30274 raft_consensus.cc:2781] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:54.964012 30274 raft_consensus.cc:482] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:54.964104 30274 raft_consensus.cc:3037] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:54.967157 30376 tablet_bootstrap.cc:492] T 48d9c7aad03d4b569c1932919d00e7d8 P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:14:54.967658 30376 ts_tablet_manager.cc:1367] T 48d9c7aad03d4b569c1932919d00e7d8 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.031s user 0.010s sys 0.014s
I20240502 14:14:54.967772 30274 raft_consensus.cc:504] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:54.969048 26819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32ea761f2834d758529890e5b496f88" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:14:54.968848 30376 raft_consensus.cc:348] T 48d9c7aad03d4b569c1932919d00e7d8 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:54.969219 30376 raft_consensus.cc:374] T 48d9c7aad03d4b569c1932919d00e7d8 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:54.969257 26819 raft_consensus.cc:3037] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:54.969332 30376 raft_consensus.cc:729] T 48d9c7aad03d4b569c1932919d00e7d8 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:14:54.971134 30376 consensus_queue.cc:260] T 48d9c7aad03d4b569c1932919d00e7d8 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:54.972026 26819 raft_consensus.cc:2454] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 1.
I20240502 14:14:54.972700 30376 ts_tablet_manager.cc:1398] T 48d9c7aad03d4b569c1932919d00e7d8 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:14:54.972963 29112 leader_election.cc:302] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:14:54.974967 30372 raft_consensus.cc:2781] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:54.975112 30372 raft_consensus.cc:686] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 LEADER]: Becoming Leader. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Running, Role: LEADER
I20240502 14:14:54.975186 29308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32ea761f2834d758529890e5b496f88" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
I20240502 14:14:54.975397 29308 raft_consensus.cc:3037] T c32ea761f2834d758529890e5b496f88 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:54.975409 30372 consensus_queue.cc:237] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:54.977901 29308 raft_consensus.cc:2454] T c32ea761f2834d758529890e5b496f88 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 1.
I20240502 14:14:54.978322 30274 leader_election.cc:288] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 election: Requested vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:14:54.988538 26276 catalog_manager.cc:5484] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197). New cstate: current_term: 1 leader_uuid: "1cb67158133e4648a6eceba32d4dbba3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:14:54.992379 30355 tablet_bootstrap.cc:492] T 48d9c7aad03d4b569c1932919d00e7d8 P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:14:54.992657 30355 ts_tablet_manager.cc:1367] T 48d9c7aad03d4b569c1932919d00e7d8 P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.042s user 0.005s sys 0.009s
I20240502 14:14:54.993605 30312 tablet_bootstrap.cc:492] T 6b4745b1153b4acd999ac3a5c137ff53 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:54.993518 30355 raft_consensus.cc:348] T 48d9c7aad03d4b569c1932919d00e7d8 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:54.993943 30355 raft_consensus.cc:374] T 48d9c7aad03d4b569c1932919d00e7d8 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:54.994062 30355 raft_consensus.cc:729] T 48d9c7aad03d4b569c1932919d00e7d8 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:14:54.994161 30312 tablet_bootstrap.cc:492] T 6b4745b1153b4acd999ac3a5c137ff53 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap complete.
I20240502 14:14:54.994383 30355 consensus_queue.cc:260] T 48d9c7aad03d4b569c1932919d00e7d8 P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:54.994501 30312 ts_tablet_manager.cc:1367] T 6b4745b1153b4acd999ac3a5c137ff53 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.584s user 0.065s sys 0.000s
I20240502 14:14:54.995302 30312 raft_consensus.cc:348] T 6b4745b1153b4acd999ac3a5c137ff53 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:54.995595 30312 raft_consensus.cc:729] T 6b4745b1153b4acd999ac3a5c137ff53 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:14:54.995978 30355 ts_tablet_manager.cc:1398] T 48d9c7aad03d4b569c1932919d00e7d8 P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:54.995893 30312 consensus_queue.cc:260] T 6b4745b1153b4acd999ac3a5c137ff53 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:54.997231 30312 ts_tablet_manager.cc:1398] T 6b4745b1153b4acd999ac3a5c137ff53 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:54.997880 26839 tablet_copy_service.cc:342] P e4cfb8fc5a474177af6848a6efa13a27: Request end of tablet copy session c1100e16b3b9495ca61d4f95e89b6a89-6b4745b1153b4acd999ac3a5c137ff53 received from {username='slave'} at 127.19.94.198:48649
I20240502 14:14:54.998019 26839 tablet_copy_service.cc:433] P e4cfb8fc5a474177af6848a6efa13a27: ending tablet copy session c1100e16b3b9495ca61d4f95e89b6a89-6b4745b1153b4acd999ac3a5c137ff53 on tablet 6b4745b1153b4acd999ac3a5c137ff53 with peer c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:14:55.006532 30362 tablet_bootstrap.cc:492] T 48d9c7aad03d4b569c1932919d00e7d8 P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:14:55.006801 30362 ts_tablet_manager.cc:1367] T 48d9c7aad03d4b569c1932919d00e7d8 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.066s user 0.008s sys 0.002s
I20240502 14:14:55.007901 30362 raft_consensus.cc:348] T 48d9c7aad03d4b569c1932919d00e7d8 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:55.008569 30362 raft_consensus.cc:374] T 48d9c7aad03d4b569c1932919d00e7d8 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:55.008726 30362 raft_consensus.cc:729] T 48d9c7aad03d4b569c1932919d00e7d8 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:14:55.009069 30362 consensus_queue.cc:260] T 48d9c7aad03d4b569c1932919d00e7d8 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:55.009756 30362 ts_tablet_manager.cc:1398] T 48d9c7aad03d4b569c1932919d00e7d8 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:14:55.082738 26423 consensus_queue.cc:237] T 058eec4efd564c348684857372c60127 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:55.085942 26557 raft_consensus.cc:1261] T 058eec4efd564c348684857372c60127 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:55.086333 26819 raft_consensus.cc:1261] T 058eec4efd564c348684857372c60127 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:55.086694 30399 consensus_queue.cc:1035] T 058eec4efd564c348684857372c60127 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:55.087086 30165 consensus_queue.cc:1035] T 058eec4efd564c348684857372c60127 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:55.092777 30399 raft_consensus.cc:2932] T 058eec4efd564c348684857372c60127 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:14:55.093673 26819 raft_consensus.cc:2932] T 058eec4efd564c348684857372c60127 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:14:55.100068 26285 catalog_manager.cc:5484] T 058eec4efd564c348684857372c60127 P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: config changed from index -1 to 4, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:14:55.101251 26557 raft_consensus.cc:2932] T 058eec4efd564c348684857372c60127 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
W20240502 14:14:55.103997 26363 consensus_peers.cc:485] T 058eec4efd564c348684857372c60127 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097): Couldn't send request to peer 4092902b3e674857a0147f1b4f2cdb86. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 058eec4efd564c348684857372c60127. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:55.131773 30177 raft_consensus.cc:482] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:55.132018 30177 raft_consensus.cc:504] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:55.133021 30177 leader_election.cc:288] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:14:55.134763 29575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0838f171f3146a181ff1f3e843cdec1" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86" is_pre_election: true
I20240502 14:14:55.135147 29575 raft_consensus.cc:2454] T d0838f171f3146a181ff1f3e843cdec1 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:14:55.135963 26625 leader_election.cc:302] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters:
I20240502 14:14:55.136607 30177 raft_consensus.cc:2781] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:55.136755 30177 raft_consensus.cc:482] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:55.136893 30177 raft_consensus.cc:3037] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:55.141809 30177 raft_consensus.cc:504] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:55.143337 30177 leader_election.cc:288] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Requested vote from peers 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:14:55.143255 29575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0838f171f3146a181ff1f3e843cdec1" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86"
I20240502 14:14:55.143563 29575 raft_consensus.cc:3037] T d0838f171f3146a181ff1f3e843cdec1 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:55.147934 29575 raft_consensus.cc:2454] T d0838f171f3146a181ff1f3e843cdec1 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:14:55.148595 26625 leader_election.cc:302] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters:
I20240502 14:14:55.148934 30177 raft_consensus.cc:2781] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:55.149515 30177 raft_consensus.cc:686] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Becoming Leader. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Running, Role: LEADER
I20240502 14:14:55.149833 30177 consensus_queue.cc:237] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:55.153789 29308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0838f171f3146a181ff1f3e843cdec1" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:14:55.153951 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0838f171f3146a181ff1f3e843cdec1" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
I20240502 14:14:55.154103 29308 raft_consensus.cc:2454] T d0838f171f3146a181ff1f3e843cdec1 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:14:55.160465 26276 catalog_manager.cc:5484] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195). New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:14:55.169713 26819 consensus_queue.cc:237] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:55.172655 26423 raft_consensus.cc:1261] T ce03ee354c54457487ea939301ccd90e P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:55.172839 26687 raft_consensus.cc:1261] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:55.173408 29791 consensus_queue.cc:1035] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:55.173772 30425 consensus_queue.cc:1035] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:14:55.174103 26755 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce03ee354c54457487ea939301ccd90e. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:55.182415 29791 raft_consensus.cc:2932] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:14:55.183279 26687 raft_consensus.cc:2932] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:14:55.187674 26423 raft_consensus.cc:2932] T ce03ee354c54457487ea939301ccd90e P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:14:55.190431 26276 catalog_manager.cc:5484] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: config changed from index -1 to 4, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:14:55.230545 29575 consensus_queue.cc:237] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
W20240502 14:14:55.233484 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): Couldn't send request to peer 69ab78b2a1b74d2bbf2cdeafa0a4af23. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1704fbfbeed04dcaac8bc663b7ee2a5b. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:55.234056 29309 raft_consensus.cc:1261] T 1704fbfbeed04dcaac8bc663b7ee2a5b P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:55.234594 29442 raft_consensus.cc:1261] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:55.235149 30089 consensus_queue.cc:1035] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:14:55.235863 30089 consensus_queue.cc:1035] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:14:55.245703 30444 raft_consensus.cc:2932] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:14:55.246356 29309 raft_consensus.cc:2932] T 1704fbfbeed04dcaac8bc663b7ee2a5b P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:14:55.253679 26285 catalog_manager.cc:5484] T 1704fbfbeed04dcaac8bc663b7ee2a5b P c1100e16b3b9495ca61d4f95e89b6a89 reported cstate change: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:14:55.277742 29442 raft_consensus.cc:2932] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:14:55.281073 30008 raft_consensus.cc:482] T 48d9c7aad03d4b569c1932919d00e7d8 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:55.281284 30008 raft_consensus.cc:504] T 48d9c7aad03d4b569c1932919d00e7d8 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:55.282100 30008 leader_election.cc:288] T 48d9c7aad03d4b569c1932919d00e7d8 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:14:55.283440 29872 raft_consensus.cc:482] T 48d9c7aad03d4b569c1932919d00e7d8 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:55.283645 29872 raft_consensus.cc:504] T 48d9c7aad03d4b569c1932919d00e7d8 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:55.284006 29575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48d9c7aad03d4b569c1932919d00e7d8" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86" is_pre_election: true
I20240502 14:14:55.284284 29575 raft_consensus.cc:2454] T 48d9c7aad03d4b569c1932919d00e7d8 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 0.
I20240502 14:14:55.284421 29872 leader_election.cc:288] T 48d9c7aad03d4b569c1932919d00e7d8 P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:14:55.284819 29379 leader_election.cc:302] T 48d9c7aad03d4b569c1932919d00e7d8 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, 68a3c2b17ed94e06aba66ed4178132ab; no voters:
I20240502 14:14:55.285154 30450 raft_consensus.cc:2781] T 48d9c7aad03d4b569c1932919d00e7d8 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:55.285269 30450 raft_consensus.cc:482] T 48d9c7aad03d4b569c1932919d00e7d8 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:55.285454 30450 raft_consensus.cc:3037] T 48d9c7aad03d4b569c1932919d00e7d8 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:55.285434 29442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48d9c7aad03d4b569c1932919d00e7d8" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:14:55.288367 29575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48d9c7aad03d4b569c1932919d00e7d8" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86" is_pre_election: true
I20240502 14:14:55.288605 30450 raft_consensus.cc:504] T 48d9c7aad03d4b569c1932919d00e7d8 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:55.288727 29575 raft_consensus.cc:2454] T 48d9c7aad03d4b569c1932919d00e7d8 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 0.
I20240502 14:14:55.288995 29442 raft_consensus.cc:2379] T 48d9c7aad03d4b569c1932919d00e7d8 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c1100e16b3b9495ca61d4f95e89b6a89 in current term 1: Already voted for candidate 68a3c2b17ed94e06aba66ed4178132ab in this term.
I20240502 14:14:55.289417 29245 leader_election.cc:302] T 48d9c7aad03d4b569c1932919d00e7d8 P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:14:55.290055 29872 raft_consensus.cc:2781] T 48d9c7aad03d4b569c1932919d00e7d8 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:55.290189 29872 raft_consensus.cc:482] T 48d9c7aad03d4b569c1932919d00e7d8 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:55.290284 29872 raft_consensus.cc:3037] T 48d9c7aad03d4b569c1932919d00e7d8 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:55.290573 29574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48d9c7aad03d4b569c1932919d00e7d8" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86"
I20240502 14:14:55.290755 29574 raft_consensus.cc:3037] T 48d9c7aad03d4b569c1932919d00e7d8 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:55.293623 29574 raft_consensus.cc:2454] T 48d9c7aad03d4b569c1932919d00e7d8 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 1.
I20240502 14:14:55.293548 29872 raft_consensus.cc:504] T 48d9c7aad03d4b569c1932919d00e7d8 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:55.294301 29379 leader_election.cc:302] T 48d9c7aad03d4b569c1932919d00e7d8 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, 68a3c2b17ed94e06aba66ed4178132ab; no voters:
I20240502 14:14:55.294656 30008 raft_consensus.cc:2781] T 48d9c7aad03d4b569c1932919d00e7d8 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:55.295445 29442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48d9c7aad03d4b569c1932919d00e7d8" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
I20240502 14:14:55.298025 29574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48d9c7aad03d4b569c1932919d00e7d8" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86"
I20240502 14:14:55.298470 29574 raft_consensus.cc:2379] T 48d9c7aad03d4b569c1932919d00e7d8 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c1100e16b3b9495ca61d4f95e89b6a89 in current term 1: Already voted for candidate 68a3c2b17ed94e06aba66ed4178132ab in this term.
I20240502 14:14:55.301870 30450 leader_election.cc:288] T 48d9c7aad03d4b569c1932919d00e7d8 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 election: Requested vote from peers 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:14:55.303642 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48d9c7aad03d4b569c1932919d00e7d8" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:14:55.304035 29309 raft_consensus.cc:2379] T 48d9c7aad03d4b569c1932919d00e7d8 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 68a3c2b17ed94e06aba66ed4178132ab in current term 1: Already voted for candidate c1100e16b3b9495ca61d4f95e89b6a89 in this term.
I20240502 14:14:55.304960 29872 leader_election.cc:288] T 48d9c7aad03d4b569c1932919d00e7d8 P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 election: Requested vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:14:55.307626 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48d9c7aad03d4b569c1932919d00e7d8" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
I20240502 14:14:55.307925 29309 raft_consensus.cc:2379] T 48d9c7aad03d4b569c1932919d00e7d8 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 68a3c2b17ed94e06aba66ed4178132ab in current term 1: Already voted for candidate c1100e16b3b9495ca61d4f95e89b6a89 in this term.
I20240502 14:14:55.307946 30008 raft_consensus.cc:686] T 48d9c7aad03d4b569c1932919d00e7d8 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 LEADER]: Becoming Leader. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Running, Role: LEADER
I20240502 14:14:55.308691 30008 consensus_queue.cc:237] T 48d9c7aad03d4b569c1932919d00e7d8 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:55.313459 29245 leader_election.cc:302] T 48d9c7aad03d4b569c1932919d00e7d8 P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c1100e16b3b9495ca61d4f95e89b6a89; no voters: 4092902b3e674857a0147f1b4f2cdb86, 68a3c2b17ed94e06aba66ed4178132ab
I20240502 14:14:55.313812 30447 raft_consensus.cc:2726] T 48d9c7aad03d4b569c1932919d00e7d8 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 14:14:55.315927 26285 catalog_manager.cc:5484] T 48d9c7aad03d4b569c1932919d00e7d8 P 68a3c2b17ed94e06aba66ed4178132ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199). New cstate: current_term: 1 leader_uuid: "68a3c2b17ed94e06aba66ed4178132ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:14:55.359674 26557 consensus_queue.cc:237] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
W20240502 14:14:55.362656 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): Couldn't send request to peer 1cb67158133e4648a6eceba32d4dbba3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 310aa6ad3604487589dbe96afa25bb05. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:55.363108 26687 raft_consensus.cc:1261] T 310aa6ad3604487589dbe96afa25bb05 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:55.363116 26819 raft_consensus.cc:1261] T 310aa6ad3604487589dbe96afa25bb05 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:55.363667 30471 consensus_queue.cc:1035] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:55.364079 30470 consensus_queue.cc:1035] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:55.366782 30470 raft_consensus.cc:2932] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:14:55.367419 26687 raft_consensus.cc:2932] T 310aa6ad3604487589dbe96afa25bb05 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:14:55.372565 26819 raft_consensus.cc:2932] T 310aa6ad3604487589dbe96afa25bb05 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:14:55.373461 26285 catalog_manager.cc:5484] T 310aa6ad3604487589dbe96afa25bb05 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: config changed from index -1 to 4, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:14:55.397100 29309 raft_consensus.cc:1203] T 6b4745b1153b4acd999ac3a5c137ff53 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:55.408324 26423 consensus_queue.cc:237] T 226b35583ee4441a82a66520aeffbb7d P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:55.410789 26557 raft_consensus.cc:1261] T 226b35583ee4441a82a66520aeffbb7d P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:55.410843 26687 raft_consensus.cc:1261] T 226b35583ee4441a82a66520aeffbb7d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:14:55.410869 26363 consensus_peers.cc:485] T 226b35583ee4441a82a66520aeffbb7d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097): Couldn't send request to peer 4092902b3e674857a0147f1b4f2cdb86. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 226b35583ee4441a82a66520aeffbb7d. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:55.411430 30165 consensus_queue.cc:1035] T 226b35583ee4441a82a66520aeffbb7d P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:55.411772 30399 consensus_queue.cc:1035] T 226b35583ee4441a82a66520aeffbb7d P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:55.414386 30165 raft_consensus.cc:2932] T 226b35583ee4441a82a66520aeffbb7d P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:14:55.415056 26557 raft_consensus.cc:2932] T 226b35583ee4441a82a66520aeffbb7d P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:14:55.425053 26285 catalog_manager.cc:5484] T 226b35583ee4441a82a66520aeffbb7d P ba187e38a2754d60a12de9ac4856430a reported cstate change: config changed from index -1 to 4, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:14:55.432263 26285 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:14:55.434878 26687 raft_consensus.cc:2932] T 226b35583ee4441a82a66520aeffbb7d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:14:55.447927 30274 consensus_queue.cc:1035] T a68c11570ad84193b88eede6c26614e3 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:55.455286 29309 raft_consensus.cc:1261] T 48d9c7aad03d4b569c1932919d00e7d8 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:55.456249 30008 consensus_queue.cc:1035] T 48d9c7aad03d4b569c1932919d00e7d8 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:55.464286 29574 raft_consensus.cc:1261] T 48d9c7aad03d4b569c1932919d00e7d8 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:55.465131 30008 consensus_queue.cc:1035] T 48d9c7aad03d4b569c1932919d00e7d8 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:55.470753 30372 consensus_queue.cc:1035] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:55.482970 26285 catalog_manager.cc:5576] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659) does not have the latest schema for tablet 48d9c7aad03d4b569c1932919d00e7d8 (table rebalancer_dynamic_table [id=d6e507d0445241a9a4bdfa70480a61a7]). Expected version 1 got 0
I20240502 14:14:55.483780 30495 mvcc.cc:205] Tried to move back new op lower bound from 7023244474175373312 to 7023244473587855360. Current Snapshot: MvccSnapshot[applied={T|T < 7023244474175373312}]
I20240502 14:14:55.504498 30508 tablet.cc:1722] T 48d9c7aad03d4b569c1932919d00e7d8 P 68a3c2b17ed94e06aba66ed4178132ab: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:55.505515 30508 tablet.cc:1620] T 48d9c7aad03d4b569c1932919d00e7d8 P 68a3c2b17ed94e06aba66ed4178132ab: MemRowSet was empty: no flush needed.
I20240502 14:14:55.505733 30511 tablet.cc:1722] T 48d9c7aad03d4b569c1932919d00e7d8 P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:55.507606 30511 tablet.cc:1620] T 48d9c7aad03d4b569c1932919d00e7d8 P c1100e16b3b9495ca61d4f95e89b6a89: MemRowSet was empty: no flush needed.
I20240502 14:14:55.509008 26285 catalog_manager.cc:5576] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659) does not have the latest schema for tablet 48d9c7aad03d4b569c1932919d00e7d8 (table rebalancer_dynamic_table [id=d6e507d0445241a9a4bdfa70480a61a7]). Expected version 1 got 0
I20240502 14:14:55.515340 30515 consensus_queue.cc:1035] T a68c11570ad84193b88eede6c26614e3 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 14:14:55.519459 26283 catalog_manager.cc:5576] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097) does not have the latest schema for tablet 48d9c7aad03d4b569c1932919d00e7d8 (table rebalancer_dynamic_table [id=d6e507d0445241a9a4bdfa70480a61a7]). Expected version 1 got 0
I20240502 14:14:55.526304 26285 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=d6e507d0445241a9a4bdfa70480a61a7] alter complete (version 1)
I20240502 14:14:55.526255 30515 consensus_queue.cc:1035] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:55.541143 26276 catalog_manager.cc:5576] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097) does not have the latest schema for tablet 48d9c7aad03d4b569c1932919d00e7d8 (table rebalancer_dynamic_table [id=d6e507d0445241a9a4bdfa70480a61a7]). Expected version 1 got 0
I20240502 14:14:55.539320 30521 tablet.cc:1722] T 48d9c7aad03d4b569c1932919d00e7d8 P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:55.544414 30521 tablet.cc:1620] T 48d9c7aad03d4b569c1932919d00e7d8 P 4092902b3e674857a0147f1b4f2cdb86: MemRowSet was empty: no flush needed.
I20240502 14:14:55.551820 30509 ts_tablet_manager.cc:902] T 058eec4efd564c348684857372c60127 P 4092902b3e674857a0147f1b4f2cdb86: Initiating tablet copy from peer 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:14:55.573863 30509 tablet_copy_client.cc:308] T 058eec4efd564c348684857372c60127 P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.193:37985
I20240502 14:14:55.574568 26447 tablet_copy_service.cc:140] P 89af7f423cf44717a08d6b142f0ed49d: Received BeginTabletCopySession request for tablet 058eec4efd564c348684857372c60127 from peer 4092902b3e674857a0147f1b4f2cdb86 ({username='slave'} at 127.19.94.200:60351)
I20240502 14:14:55.574730 26447 tablet_copy_service.cc:161] P 89af7f423cf44717a08d6b142f0ed49d: Beginning new tablet copy session on tablet 058eec4efd564c348684857372c60127 from peer 4092902b3e674857a0147f1b4f2cdb86 at {username='slave'} at 127.19.94.200:60351: session id = 4092902b3e674857a0147f1b4f2cdb86-058eec4efd564c348684857372c60127
I20240502 14:14:55.577771 26447 tablet_copy_source_session.cc:205] T 058eec4efd564c348684857372c60127 P 89af7f423cf44717a08d6b142f0ed49d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:55.579195 30509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 058eec4efd564c348684857372c60127. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:55.584897 30509 tablet_copy_client.cc:783] T 058eec4efd564c348684857372c60127 P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:55.585209 30509 tablet_copy_client.cc:647] T 058eec4efd564c348684857372c60127 P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:55.587024 30509 tablet_copy_client.cc:523] T 058eec4efd564c348684857372c60127 P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:55.590309 30509 tablet_bootstrap.cc:492] T 058eec4efd564c348684857372c60127 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:14:55.596772 29575 consensus_queue.cc:237] T 2532ebd7bdf14876b74a26afd674a7ac P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:55.597051 30534 ts_tablet_manager.cc:902] T ce03ee354c54457487ea939301ccd90e P 68a3c2b17ed94e06aba66ed4178132ab: Initiating tablet copy from peer e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
W20240502 14:14:55.607627 29510 consensus_peers.cc:485] T 2532ebd7bdf14876b74a26afd674a7ac P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): Couldn't send request to peer 89af7f423cf44717a08d6b142f0ed49d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2532ebd7bdf14876b74a26afd674a7ac. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:55.608211 29442 raft_consensus.cc:1261] T 2532ebd7bdf14876b74a26afd674a7ac P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:55.608919 30444 consensus_queue.cc:1035] T 2532ebd7bdf14876b74a26afd674a7ac P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:14:55.621752 29174 raft_consensus.cc:1261] T 2532ebd7bdf14876b74a26afd674a7ac P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:55.622601 30444 consensus_queue.cc:1035] T 2532ebd7bdf14876b74a26afd674a7ac P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:14:55.627362 30089 raft_consensus.cc:2932] T 2532ebd7bdf14876b74a26afd674a7ac P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:14:55.629057 29174 raft_consensus.cc:2932] T 2532ebd7bdf14876b74a26afd674a7ac P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:14:55.633176 29443 raft_consensus.cc:2932] T 2532ebd7bdf14876b74a26afd674a7ac P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:14:55.633977 30534 tablet_copy_client.cc:308] T ce03ee354c54457487ea939301ccd90e P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.196:32843
I20240502 14:14:55.636165 26285 catalog_manager.cc:5484] T 2532ebd7bdf14876b74a26afd674a7ac P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: config changed from index -1 to 2, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:14:55.644299 26276 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:14:55.660115 30508 tablet.cc:1722] T 48d9c7aad03d4b569c1932919d00e7d8 P 68a3c2b17ed94e06aba66ed4178132ab: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:55.660796 30508 tablet.cc:1620] T 48d9c7aad03d4b569c1932919d00e7d8 P 68a3c2b17ed94e06aba66ed4178132ab: MemRowSet was empty: no flush needed.
I20240502 14:14:55.661737 30511 tablet.cc:1722] T 48d9c7aad03d4b569c1932919d00e7d8 P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:55.662451 30511 tablet.cc:1620] T 48d9c7aad03d4b569c1932919d00e7d8 P c1100e16b3b9495ca61d4f95e89b6a89: MemRowSet was empty: no flush needed.
I20240502 14:14:55.664485 30521 tablet.cc:1722] T 48d9c7aad03d4b569c1932919d00e7d8 P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:55.665002 30521 tablet.cc:1620] T 48d9c7aad03d4b569c1932919d00e7d8 P 4092902b3e674857a0147f1b4f2cdb86: MemRowSet was empty: no flush needed.
I20240502 14:14:55.673964 26839 tablet_copy_service.cc:140] P e4cfb8fc5a474177af6848a6efa13a27: Received BeginTabletCopySession request for tablet ce03ee354c54457487ea939301ccd90e from peer 68a3c2b17ed94e06aba66ed4178132ab ({username='slave'} at 127.19.94.199:38699)
I20240502 14:14:55.674153 26839 tablet_copy_service.cc:161] P e4cfb8fc5a474177af6848a6efa13a27: Beginning new tablet copy session on tablet ce03ee354c54457487ea939301ccd90e from peer 68a3c2b17ed94e06aba66ed4178132ab at {username='slave'} at 127.19.94.199:38699: session id = 68a3c2b17ed94e06aba66ed4178132ab-ce03ee354c54457487ea939301ccd90e
I20240502 14:14:55.677405 26839 tablet_copy_source_session.cc:205] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:55.681448 30534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce03ee354c54457487ea939301ccd90e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:55.682858 26285 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=d6e507d0445241a9a4bdfa70480a61a7] alter complete (version 2)
I20240502 14:14:55.687371 30509 tablet_bootstrap.cc:492] T 058eec4efd564c348684857372c60127 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:55.687844 30509 tablet_bootstrap.cc:492] T 058eec4efd564c348684857372c60127 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap complete.
I20240502 14:14:55.688158 30509 ts_tablet_manager.cc:1367] T 058eec4efd564c348684857372c60127 P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.098s user 0.060s sys 0.003s
I20240502 14:14:55.689047 30509 raft_consensus.cc:348] T 058eec4efd564c348684857372c60127 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:55.689304 30509 raft_consensus.cc:729] T 058eec4efd564c348684857372c60127 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:14:55.689549 30509 consensus_queue.cc:260] T 058eec4efd564c348684857372c60127 P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:55.690129 30509 ts_tablet_manager.cc:1398] T 058eec4efd564c348684857372c60127 P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240502 14:14:55.690773 26447 tablet_copy_service.cc:342] P 89af7f423cf44717a08d6b142f0ed49d: Request end of tablet copy session 4092902b3e674857a0147f1b4f2cdb86-058eec4efd564c348684857372c60127 received from {username='slave'} at 127.19.94.200:60351
I20240502 14:14:55.690939 26447 tablet_copy_service.cc:433] P 89af7f423cf44717a08d6b142f0ed49d: ending tablet copy session 4092902b3e674857a0147f1b4f2cdb86-058eec4efd564c348684857372c60127 on tablet 058eec4efd564c348684857372c60127 with peer 4092902b3e674857a0147f1b4f2cdb86
I20240502 14:14:55.691622 30534 tablet_copy_client.cc:783] T ce03ee354c54457487ea939301ccd90e P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:55.692027 30534 tablet_copy_client.cc:647] T ce03ee354c54457487ea939301ccd90e P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:55.696593 30534 tablet_copy_client.cc:523] T ce03ee354c54457487ea939301ccd90e P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:55.701010 30534 tablet_bootstrap.cc:492] T ce03ee354c54457487ea939301ccd90e P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:14:55.717312 26276 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_extra_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:55.730814 29154 tablet_service.cc:1467] Processing CreateTablet for tablet f4973d1d77904cb9a3cf8d49346b330e (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=312a6520544b4149a5813f3f7652f51f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:55.731302 29154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4973d1d77904cb9a3cf8d49346b330e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:55.734473 29288 tablet_service.cc:1467] Processing CreateTablet for tablet f4973d1d77904cb9a3cf8d49346b330e (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=312a6520544b4149a5813f3f7652f51f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:55.734961 29288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4973d1d77904cb9a3cf8d49346b330e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:55.741015 26668 tablet_service.cc:1467] Processing CreateTablet for tablet f4973d1d77904cb9a3cf8d49346b330e (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=312a6520544b4149a5813f3f7652f51f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:55.741197 29553 tablet_service.cc:1467] Processing CreateTablet for tablet 82e51aa5371e4731bc00daafc65c7a88 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=312a6520544b4149a5813f3f7652f51f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:55.741536 26668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4973d1d77904cb9a3cf8d49346b330e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:55.741611 29553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82e51aa5371e4731bc00daafc65c7a88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:55.744786 29554 tablet_service.cc:1467] Processing CreateTablet for tablet 0d0298cfb57342f7b840a8656f452645 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=312a6520544b4149a5813f3f7652f51f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:55.745049 29423 tablet_service.cc:1467] Processing CreateTablet for tablet 82e51aa5371e4731bc00daafc65c7a88 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=312a6520544b4149a5813f3f7652f51f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:55.745067 26797 tablet_service.cc:1467] Processing CreateTablet for tablet 0d0298cfb57342f7b840a8656f452645 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=312a6520544b4149a5813f3f7652f51f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:55.745221 29554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d0298cfb57342f7b840a8656f452645. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:55.745478 26797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d0298cfb57342f7b840a8656f452645. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:55.745489 29423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82e51aa5371e4731bc00daafc65c7a88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:55.748811 29422 tablet_service.cc:1467] Processing CreateTablet for tablet 0d0298cfb57342f7b840a8656f452645 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=312a6520544b4149a5813f3f7652f51f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:55.749249 29422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d0298cfb57342f7b840a8656f452645. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:55.752862 29308 raft_consensus.cc:3037] T d0838f171f3146a181ff1f3e843cdec1 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:55.752959 29156 tablet_service.cc:1467] Processing CreateTablet for tablet 82e51aa5371e4731bc00daafc65c7a88 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=312a6520544b4149a5813f3f7652f51f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:55.753377 29156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82e51aa5371e4731bc00daafc65c7a88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:55.759445 30417 consensus_queue.cc:1035] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:55.763309 30177 consensus_queue.cc:1035] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:55.770180 30534 tablet_bootstrap.cc:492] T ce03ee354c54457487ea939301ccd90e P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:55.770699 30534 tablet_bootstrap.cc:492] T ce03ee354c54457487ea939301ccd90e P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap complete.
I20240502 14:14:55.771085 30534 ts_tablet_manager.cc:1367] T ce03ee354c54457487ea939301ccd90e P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.070s user 0.047s sys 0.011s
I20240502 14:14:55.781533 30534 raft_consensus.cc:348] T ce03ee354c54457487ea939301ccd90e P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:55.781904 30534 raft_consensus.cc:729] T ce03ee354c54457487ea939301ccd90e P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:14:55.782227 30534 consensus_queue.cc:260] T ce03ee354c54457487ea939301ccd90e P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:55.782971 30534 ts_tablet_manager.cc:1398] T ce03ee354c54457487ea939301ccd90e P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20240502 14:14:55.784027 30555 tablet_bootstrap.cc:492] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:14:55.785465 26839 tablet_copy_service.cc:342] P e4cfb8fc5a474177af6848a6efa13a27: Request end of tablet copy session 68a3c2b17ed94e06aba66ed4178132ab-ce03ee354c54457487ea939301ccd90e received from {username='slave'} at 127.19.94.199:38699
I20240502 14:14:55.785632 26839 tablet_copy_service.cc:433] P e4cfb8fc5a474177af6848a6efa13a27: ending tablet copy session 68a3c2b17ed94e06aba66ed4178132ab-ce03ee354c54457487ea939301ccd90e on tablet ce03ee354c54457487ea939301ccd90e with peer 68a3c2b17ed94e06aba66ed4178132ab
I20240502 14:14:55.786314 30555 tablet_bootstrap.cc:654] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:55.791965 26276 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:14:55.796517 30555 tablet_bootstrap.cc:492] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:14:55.796968 30555 ts_tablet_manager.cc:1367] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20240502 14:14:55.798148 30555 raft_consensus.cc:348] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:55.798547 30555 raft_consensus.cc:374] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:55.798699 30555 raft_consensus.cc:729] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:14:55.799033 30555 consensus_queue.cc:260] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:55.799750 30555 ts_tablet_manager.cc:1398] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:55.800716 30555 tablet_bootstrap.cc:492] T 82e51aa5371e4731bc00daafc65c7a88 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:14:55.802500 30555 tablet_bootstrap.cc:654] T 82e51aa5371e4731bc00daafc65c7a88 P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:55.804658 30556 tablet_bootstrap.cc:492] T f4973d1d77904cb9a3cf8d49346b330e P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:14:55.805203 30508 tablet.cc:1722] T 48d9c7aad03d4b569c1932919d00e7d8 P 68a3c2b17ed94e06aba66ed4178132ab: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:55.805969 30564 tablet_bootstrap.cc:492] T 0d0298cfb57342f7b840a8656f452645 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:14:55.806417 30556 tablet_bootstrap.cc:654] T f4973d1d77904cb9a3cf8d49346b330e P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:55.808176 30564 tablet_bootstrap.cc:654] T 0d0298cfb57342f7b840a8656f452645 P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:55.812551 30556 tablet_bootstrap.cc:492] T f4973d1d77904cb9a3cf8d49346b330e P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:14:55.813143 30556 ts_tablet_manager.cc:1367] T f4973d1d77904cb9a3cf8d49346b330e P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240502 14:14:55.815001 30556 raft_consensus.cc:348] T f4973d1d77904cb9a3cf8d49346b330e P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:55.816206 30556 raft_consensus.cc:374] T f4973d1d77904cb9a3cf8d49346b330e P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:55.816502 30556 raft_consensus.cc:729] T f4973d1d77904cb9a3cf8d49346b330e P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:14:55.816411 30521 tablet.cc:1722] T 48d9c7aad03d4b569c1932919d00e7d8 P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:55.817274 30560 tablet_bootstrap.cc:492] T 0d0298cfb57342f7b840a8656f452645 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:14:55.820468 30557 tablet_bootstrap.cc:492] T 0d0298cfb57342f7b840a8656f452645 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:55.819242 30556 consensus_queue.cc:260] T f4973d1d77904cb9a3cf8d49346b330e P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:55.821565 30556 ts_tablet_manager.cc:1398] T f4973d1d77904cb9a3cf8d49346b330e P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 14:14:55.823798 30557 tablet_bootstrap.cc:654] T 0d0298cfb57342f7b840a8656f452645 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:55.824311 30560 tablet_bootstrap.cc:654] T 0d0298cfb57342f7b840a8656f452645 P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:55.825110 30511 tablet.cc:1722] T 48d9c7aad03d4b569c1932919d00e7d8 P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:55.834275 30557 tablet_bootstrap.cc:492] T 0d0298cfb57342f7b840a8656f452645 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:55.834573 30557 ts_tablet_manager.cc:1367] T 0d0298cfb57342f7b840a8656f452645 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.002s
I20240502 14:14:55.835865 30557 raft_consensus.cc:348] T 0d0298cfb57342f7b840a8656f452645 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:55.838055 30557 raft_consensus.cc:374] T 0d0298cfb57342f7b840a8656f452645 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:55.838331 30557 raft_consensus.cc:729] T 0d0298cfb57342f7b840a8656f452645 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:55.838809 30557 consensus_queue.cc:260] T 0d0298cfb57342f7b840a8656f452645 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:55.841048 30557 ts_tablet_manager.cc:1398] T 0d0298cfb57342f7b840a8656f452645 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20240502 14:14:55.842564 30560 tablet_bootstrap.cc:492] T 0d0298cfb57342f7b840a8656f452645 P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:14:55.843371 30560 ts_tablet_manager.cc:1367] T 0d0298cfb57342f7b840a8656f452645 P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.006s
I20240502 14:14:55.845100 30560 raft_consensus.cc:348] T 0d0298cfb57342f7b840a8656f452645 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:55.845973 30560 raft_consensus.cc:374] T 0d0298cfb57342f7b840a8656f452645 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:55.846262 30560 raft_consensus.cc:729] T 0d0298cfb57342f7b840a8656f452645 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:14:55.846807 30560 consensus_queue.cc:260] T 0d0298cfb57342f7b840a8656f452645 P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:55.848738 30560 ts_tablet_manager.cc:1398] T 0d0298cfb57342f7b840a8656f452645 P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 14:14:55.849570 30560 tablet_bootstrap.cc:492] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:14:55.853873 30425 raft_consensus.cc:482] T 0d0298cfb57342f7b840a8656f452645 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:55.854149 30425 raft_consensus.cc:504] T 0d0298cfb57342f7b840a8656f452645 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:55.855113 30425 leader_election.cc:288] T 0d0298cfb57342f7b840a8656f452645 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:14:55.856199 30560 tablet_bootstrap.cc:654] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:55.856766 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d0298cfb57342f7b840a8656f452645" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:14:55.857795 29575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d0298cfb57342f7b840a8656f452645" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86" is_pre_election: true
I20240502 14:14:55.858057 29575 raft_consensus.cc:2454] T 0d0298cfb57342f7b840a8656f452645 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 0.
I20240502 14:14:55.858783 26755 leader_election.cc:302] T 0d0298cfb57342f7b840a8656f452645 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, e4cfb8fc5a474177af6848a6efa13a27; no voters:
W20240502 14:14:55.859195 26755 leader_election.cc:341] T 0d0298cfb57342f7b840a8656f452645 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:55.859953 30425 raft_consensus.cc:2781] T 0d0298cfb57342f7b840a8656f452645 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:55.860073 30425 raft_consensus.cc:482] T 0d0298cfb57342f7b840a8656f452645 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:55.860188 30425 raft_consensus.cc:3037] T 0d0298cfb57342f7b840a8656f452645 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:55.876180 30425 raft_consensus.cc:504] T 0d0298cfb57342f7b840a8656f452645 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:55.876965 26285 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table_new_name [id=d6e507d0445241a9a4bdfa70480a61a7] alter complete (version 3)
I20240502 14:14:55.885790 29575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d0298cfb57342f7b840a8656f452645" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86"
I20240502 14:14:55.886102 29575 raft_consensus.cc:3037] T 0d0298cfb57342f7b840a8656f452645 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:55.895802 29575 raft_consensus.cc:2454] T 0d0298cfb57342f7b840a8656f452645 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
I20240502 14:14:55.899171 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d0298cfb57342f7b840a8656f452645" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
W20240502 14:14:55.900079 26755 leader_election.cc:341] T 0d0298cfb57342f7b840a8656f452645 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:55.900863 30425 leader_election.cc:288] T 0d0298cfb57342f7b840a8656f452645 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Requested vote from peers 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:14:55.902105 26755 leader_election.cc:302] T 0d0298cfb57342f7b840a8656f452645 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab
I20240502 14:14:55.902909 30425 raft_consensus.cc:2781] T 0d0298cfb57342f7b840a8656f452645 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:55.904201 30425 raft_consensus.cc:686] T 0d0298cfb57342f7b840a8656f452645 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Becoming Leader. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Running, Role: LEADER
I20240502 14:14:55.904887 30425 consensus_queue.cc:237] T 0d0298cfb57342f7b840a8656f452645 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:55.921705 30568 ts_tablet_manager.cc:902] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Initiating tablet copy from peer 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:14:55.925294 30565 tablet_bootstrap.cc:492] T f4973d1d77904cb9a3cf8d49346b330e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:55.929795 26276 catalog_manager.cc:5484] T 0d0298cfb57342f7b840a8656f452645 P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196). New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:14:55.940977 30274 raft_consensus.cc:482] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:55.941268 30274 raft_consensus.cc:504] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:55.942236 30274 leader_election.cc:288] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:14:55.943302 26689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4973d1d77904cb9a3cf8d49346b330e" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:14:55.943331 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4973d1d77904cb9a3cf8d49346b330e" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:14:55.943658 29309 raft_consensus.cc:2454] T f4973d1d77904cb9a3cf8d49346b330e P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 0.
W20240502 14:14:55.944849 29112 leader_election.cc:341] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:55.945633 29111 leader_election.cc:302] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, c1100e16b3b9495ca61d4f95e89b6a89; no voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23
I20240502 14:14:55.949642 30274 raft_consensus.cc:2781] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:55.949800 30274 raft_consensus.cc:482] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:55.949934 30274 raft_consensus.cc:3037] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:55.960266 30274 raft_consensus.cc:504] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:55.962531 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4973d1d77904cb9a3cf8d49346b330e" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
I20240502 14:14:55.962827 29309 raft_consensus.cc:3037] T f4973d1d77904cb9a3cf8d49346b330e P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:55.966398 30580 ts_tablet_manager.cc:902] T 310aa6ad3604487589dbe96afa25bb05 P 1cb67158133e4648a6eceba32d4dbba3: Initiating tablet copy from peer ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:14:55.981565 29309 raft_consensus.cc:2454] T f4973d1d77904cb9a3cf8d49346b330e P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 1.
I20240502 14:14:55.982241 29111 leader_election.cc:302] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:14:55.982707 30515 raft_consensus.cc:2781] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:55.982894 30515 raft_consensus.cc:686] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 [term 1 LEADER]: Becoming Leader. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Running, Role: LEADER
I20240502 14:14:55.983227 30515 consensus_queue.cc:237] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:55.986451 26283 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:14:55.986778 26283 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:14:55.990617 30509 ts_tablet_manager.cc:902] T 226b35583ee4441a82a66520aeffbb7d P 4092902b3e674857a0147f1b4f2cdb86: Initiating tablet copy from peer 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:14:55.995383 26283 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet 48d9c7aad03d4b569c1932919d00e7d8
I20240502 14:14:55.997604 29422 tablet_service.cc:1514] Processing DeleteTablet for tablet 48d9c7aad03d4b569c1932919d00e7d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:55 UTC) from {username='slave'} at 127.0.0.1:40652
I20240502 14:14:55.998502 29554 tablet_service.cc:1514] Processing DeleteTablet for tablet 48d9c7aad03d4b569c1932919d00e7d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:55 UTC) from {username='slave'} at 127.0.0.1:36398
I20240502 14:14:55.999295 29288 tablet_service.cc:1514] Processing DeleteTablet for tablet 48d9c7aad03d4b569c1932919d00e7d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:55 UTC) from {username='slave'} at 127.0.0.1:59160
I20240502 14:14:56.010962 30588 tablet_replica.cc:330] T 48d9c7aad03d4b569c1932919d00e7d8 P 4092902b3e674857a0147f1b4f2cdb86: stopping tablet replica
I20240502 14:14:56.010700 30274 leader_election.cc:288] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 election: Requested vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:14:55.998293 30587 tablet_replica.cc:330] T 48d9c7aad03d4b569c1932919d00e7d8 P 68a3c2b17ed94e06aba66ed4178132ab: stopping tablet replica
I20240502 14:14:56.011538 30587 raft_consensus.cc:2229] T 48d9c7aad03d4b569c1932919d00e7d8 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:56.011878 30588 raft_consensus.cc:2229] T 48d9c7aad03d4b569c1932919d00e7d8 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:56.012495 30587 raft_consensus.cc:2258] T 48d9c7aad03d4b569c1932919d00e7d8 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:56.012871 30588 raft_consensus.cc:2258] T 48d9c7aad03d4b569c1932919d00e7d8 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:56.014577 30588 ts_tablet_manager.cc:1875] T 48d9c7aad03d4b569c1932919d00e7d8 P 4092902b3e674857a0147f1b4f2cdb86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:56.021188 30587 ts_tablet_manager.cc:1875] T 48d9c7aad03d4b569c1932919d00e7d8 P 68a3c2b17ed94e06aba66ed4178132ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:56.028084 30580 tablet_copy_client.cc:308] T 310aa6ad3604487589dbe96afa25bb05 P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.194:36025
I20240502 14:14:56.028997 26578 tablet_copy_service.cc:140] P ba187e38a2754d60a12de9ac4856430a: Received BeginTabletCopySession request for tablet 310aa6ad3604487589dbe96afa25bb05 from peer 1cb67158133e4648a6eceba32d4dbba3 ({username='slave'} at 127.19.94.197:58769)
I20240502 14:14:56.029196 26578 tablet_copy_service.cc:161] P ba187e38a2754d60a12de9ac4856430a: Beginning new tablet copy session on tablet 310aa6ad3604487589dbe96afa25bb05 from peer 1cb67158133e4648a6eceba32d4dbba3 at {username='slave'} at 127.19.94.197:58769: session id = 1cb67158133e4648a6eceba32d4dbba3-310aa6ad3604487589dbe96afa25bb05
I20240502 14:14:56.032225 26578 tablet_copy_source_session.cc:205] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:56.035291 30580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 310aa6ad3604487589dbe96afa25bb05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:56.036667 30565 tablet_bootstrap.cc:654] T f4973d1d77904cb9a3cf8d49346b330e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:56.037101 26687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4973d1d77904cb9a3cf8d49346b330e" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
W20240502 14:14:56.037627 29112 leader_election.cc:341] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:56.046000 30587 ts_tablet_manager.cc:1888] T 48d9c7aad03d4b569c1932919d00e7d8 P 68a3c2b17ed94e06aba66ed4178132ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:56.048543 30588 ts_tablet_manager.cc:1888] T 48d9c7aad03d4b569c1932919d00e7d8 P 4092902b3e674857a0147f1b4f2cdb86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:56.048874 30587 log.cc:1196] T 48d9c7aad03d4b569c1932919d00e7d8 P 68a3c2b17ed94e06aba66ed4178132ab: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal/wals/48d9c7aad03d4b569c1932919d00e7d8
I20240502 14:14:56.049614 30587 ts_tablet_manager.cc:1909] T 48d9c7aad03d4b569c1932919d00e7d8 P 68a3c2b17ed94e06aba66ed4178132ab: Deleting consensus metadata
I20240502 14:14:56.049983 30588 log.cc:1196] T 48d9c7aad03d4b569c1932919d00e7d8 P 4092902b3e674857a0147f1b4f2cdb86: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/wal/wals/48d9c7aad03d4b569c1932919d00e7d8
I20240502 14:14:56.050675 30588 ts_tablet_manager.cc:1909] T 48d9c7aad03d4b569c1932919d00e7d8 P 4092902b3e674857a0147f1b4f2cdb86: Deleting consensus metadata
I20240502 14:14:56.052143 26272 catalog_manager.cc:4830] TS 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): tablet 48d9c7aad03d4b569c1932919d00e7d8 (table rebalancer_dynamic_table_new_name [id=d6e507d0445241a9a4bdfa70480a61a7]) successfully deleted
I20240502 14:14:56.053102 30589 tablet_replica.cc:330] T 48d9c7aad03d4b569c1932919d00e7d8 P c1100e16b3b9495ca61d4f95e89b6a89: stopping tablet replica
I20240502 14:14:56.053387 30589 raft_consensus.cc:2229] T 48d9c7aad03d4b569c1932919d00e7d8 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:56.053681 30589 raft_consensus.cc:2258] T 48d9c7aad03d4b569c1932919d00e7d8 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:56.052245 30509 tablet_copy_client.cc:308] T 226b35583ee4441a82a66520aeffbb7d P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.193:37985
I20240502 14:14:56.055045 26447 tablet_copy_service.cc:140] P 89af7f423cf44717a08d6b142f0ed49d: Received BeginTabletCopySession request for tablet 226b35583ee4441a82a66520aeffbb7d from peer 4092902b3e674857a0147f1b4f2cdb86 ({username='slave'} at 127.19.94.200:60351)
I20240502 14:14:56.055238 26447 tablet_copy_service.cc:161] P 89af7f423cf44717a08d6b142f0ed49d: Beginning new tablet copy session on tablet 226b35583ee4441a82a66520aeffbb7d from peer 4092902b3e674857a0147f1b4f2cdb86 at {username='slave'} at 127.19.94.200:60351: session id = 4092902b3e674857a0147f1b4f2cdb86-226b35583ee4441a82a66520aeffbb7d
I20240502 14:14:56.055616 26272 catalog_manager.cc:4830] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097): tablet 48d9c7aad03d4b569c1932919d00e7d8 (table rebalancer_dynamic_table_new_name [id=d6e507d0445241a9a4bdfa70480a61a7]) successfully deleted
I20240502 14:14:56.058526 26447 tablet_copy_source_session.cc:205] T 226b35583ee4441a82a66520aeffbb7d P 89af7f423cf44717a08d6b142f0ed49d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:56.060060 30509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 226b35583ee4441a82a66520aeffbb7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:56.065706 30580 tablet_copy_client.cc:783] T 310aa6ad3604487589dbe96afa25bb05 P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:56.066453 30580 tablet_copy_client.cc:647] T 310aa6ad3604487589dbe96afa25bb05 P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:56.070055 30580 tablet_copy_client.cc:523] T 310aa6ad3604487589dbe96afa25bb05 P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:56.072710 26276 catalog_manager.cc:5484] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197). New cstate: current_term: 1 leader_uuid: "1cb67158133e4648a6eceba32d4dbba3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:14:56.084074 30589 ts_tablet_manager.cc:1875] T 48d9c7aad03d4b569c1932919d00e7d8 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:56.085677 30509 tablet_copy_client.cc:783] T 226b35583ee4441a82a66520aeffbb7d P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:56.085935 30509 tablet_copy_client.cc:647] T 226b35583ee4441a82a66520aeffbb7d P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:56.089066 30509 tablet_copy_client.cc:523] T 226b35583ee4441a82a66520aeffbb7d P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:56.089377 29575 raft_consensus.cc:1203] T 058eec4efd564c348684857372c60127 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:56.095238 30560 tablet_bootstrap.cc:492] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:14:56.095444 30560 ts_tablet_manager.cc:1367] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.246s user 0.052s sys 0.079s
I20240502 14:14:56.096282 30560 raft_consensus.cc:348] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:56.096546 30560 raft_consensus.cc:374] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:56.096643 30560 raft_consensus.cc:729] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:14:56.096961 30560 consensus_queue.cc:260] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:56.098415 30560 ts_tablet_manager.cc:1398] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 14:14:56.100982 30555 tablet_bootstrap.cc:492] T 82e51aa5371e4731bc00daafc65c7a88 P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:14:56.104785 30555 ts_tablet_manager.cc:1367] T 82e51aa5371e4731bc00daafc65c7a88 P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.304s user 0.054s sys 0.109s
I20240502 14:14:56.105688 30555 raft_consensus.cc:348] T 82e51aa5371e4731bc00daafc65c7a88 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:56.105938 30555 raft_consensus.cc:374] T 82e51aa5371e4731bc00daafc65c7a88 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:56.106031 30555 raft_consensus.cc:729] T 82e51aa5371e4731bc00daafc65c7a88 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:14:56.106298 30555 consensus_queue.cc:260] T 82e51aa5371e4731bc00daafc65c7a88 P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:56.106986 30555 ts_tablet_manager.cc:1398] T 82e51aa5371e4731bc00daafc65c7a88 P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:14:56.088008 30580 tablet_bootstrap.cc:492] T 310aa6ad3604487589dbe96afa25bb05 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:14:56.111465 30565 tablet_bootstrap.cc:492] T f4973d1d77904cb9a3cf8d49346b330e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:56.111632 30565 ts_tablet_manager.cc:1367] T f4973d1d77904cb9a3cf8d49346b330e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.186s user 0.002s sys 0.003s
I20240502 14:14:56.112517 30509 tablet_bootstrap.cc:492] T 226b35583ee4441a82a66520aeffbb7d P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:14:56.112495 30565 raft_consensus.cc:348] T f4973d1d77904cb9a3cf8d49346b330e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:56.112730 30565 raft_consensus.cc:374] T f4973d1d77904cb9a3cf8d49346b330e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:56.112829 30565 raft_consensus.cc:729] T f4973d1d77904cb9a3cf8d49346b330e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:56.113086 30565 consensus_queue.cc:260] T f4973d1d77904cb9a3cf8d49346b330e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:56.113672 30565 ts_tablet_manager.cc:1398] T f4973d1d77904cb9a3cf8d49346b330e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:56.116297 29443 raft_consensus.cc:1203] T ce03ee354c54457487ea939301ccd90e P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:56.117929 26557 consensus_queue.cc:237] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:56.148880 26283 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:14:56.165454 29554 tablet_service.cc:1467] Processing CreateTablet for tablet 9eb6af02e761412dae5ab888a4b97455 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ec92b676b3e04dcc8943814a451c0c4a]), partition=
I20240502 14:14:56.165899 29554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eb6af02e761412dae5ab888a4b97455. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:56.166265 29288 tablet_service.cc:1467] Processing CreateTablet for tablet 9eb6af02e761412dae5ab888a4b97455 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ec92b676b3e04dcc8943814a451c0c4a]), partition=
I20240502 14:14:56.166307 26406 tablet_service.cc:1467] Processing CreateTablet for tablet 9eb6af02e761412dae5ab888a4b97455 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ec92b676b3e04dcc8943814a451c0c4a]), partition=
I20240502 14:14:56.166666 26406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eb6af02e761412dae5ab888a4b97455. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:56.166774 29288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eb6af02e761412dae5ab888a4b97455. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:56.171021 30564 tablet_bootstrap.cc:492] T 0d0298cfb57342f7b840a8656f452645 P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:14:56.171190 30564 ts_tablet_manager.cc:1367] T 0d0298cfb57342f7b840a8656f452645 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.365s user 0.049s sys 0.049s
I20240502 14:14:56.172117 30564 raft_consensus.cc:348] T 0d0298cfb57342f7b840a8656f452645 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:56.172331 30564 raft_consensus.cc:374] T 0d0298cfb57342f7b840a8656f452645 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:56.172456 30564 raft_consensus.cc:729] T 0d0298cfb57342f7b840a8656f452645 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:14:56.172693 30564 consensus_queue.cc:260] T 0d0298cfb57342f7b840a8656f452645 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:56.173900 30556 tablet_bootstrap.cc:492] T 9eb6af02e761412dae5ab888a4b97455 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:14:56.175668 30556 tablet_bootstrap.cc:654] T 9eb6af02e761412dae5ab888a4b97455 P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:56.177951 30564 ts_tablet_manager.cc:1398] T 0d0298cfb57342f7b840a8656f452645 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:14:56.178277 30564 tablet_bootstrap.cc:492] T 82e51aa5371e4731bc00daafc65c7a88 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:14:56.180853 30560 tablet_bootstrap.cc:492] T 9eb6af02e761412dae5ab888a4b97455 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:14:56.181403 30564 tablet_bootstrap.cc:654] T 82e51aa5371e4731bc00daafc65c7a88 P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:56.182525 30560 tablet_bootstrap.cc:654] T 9eb6af02e761412dae5ab888a4b97455 P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:56.196166 30589 ts_tablet_manager.cc:1888] T 48d9c7aad03d4b569c1932919d00e7d8 P c1100e16b3b9495ca61d4f95e89b6a89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:56.210770 30601 ts_tablet_manager.cc:902] T 2532ebd7bdf14876b74a26afd674a7ac P 89af7f423cf44717a08d6b142f0ed49d: Initiating tablet copy from peer 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:14:56.213495 26687 raft_consensus.cc:1261] T d7f02f897ce54fefb03832da3c2d95e0 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:56.214056 26819 raft_consensus.cc:1261] T d7f02f897ce54fefb03832da3c2d95e0 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:56.214709 30568 tablet_copy_client.cc:308] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.200:46097
I20240502 14:14:56.215811 29595 tablet_copy_service.cc:140] P 4092902b3e674857a0147f1b4f2cdb86: Received BeginTabletCopySession request for tablet 1704fbfbeed04dcaac8bc663b7ee2a5b from peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 ({username='slave'} at 127.19.94.195:39709)
I20240502 14:14:56.215992 29595 tablet_copy_service.cc:161] P 4092902b3e674857a0147f1b4f2cdb86: Beginning new tablet copy session on tablet 1704fbfbeed04dcaac8bc663b7ee2a5b from peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 at {username='slave'} at 127.19.94.195:39709: session id = 69ab78b2a1b74d2bbf2cdeafa0a4af23-1704fbfbeed04dcaac8bc663b7ee2a5b
I20240502 14:14:56.218196 30471 consensus_queue.cc:1035] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:56.218799 30471 consensus_queue.cc:1035] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:56.219534 29595 tablet_copy_source_session.cc:205] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:56.220688 30589 log.cc:1196] T 48d9c7aad03d4b569c1932919d00e7d8 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal/wals/48d9c7aad03d4b569c1932919d00e7d8
I20240502 14:14:56.221521 30589 ts_tablet_manager.cc:1909] T 48d9c7aad03d4b569c1932919d00e7d8 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting consensus metadata
I20240502 14:14:56.222550 30568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1704fbfbeed04dcaac8bc663b7ee2a5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:56.226553 26271 catalog_manager.cc:4830] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): tablet 48d9c7aad03d4b569c1932919d00e7d8 (table rebalancer_dynamic_table_new_name [id=d6e507d0445241a9a4bdfa70480a61a7]) successfully deleted
I20240502 14:14:56.227828 30606 tablet_bootstrap.cc:492] T 9eb6af02e761412dae5ab888a4b97455 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:56.233369 30606 tablet_bootstrap.cc:654] T 9eb6af02e761412dae5ab888a4b97455 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:56.237295 30556 tablet_bootstrap.cc:492] T 9eb6af02e761412dae5ab888a4b97455 P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:14:56.237581 30556 ts_tablet_manager.cc:1367] T 9eb6af02e761412dae5ab888a4b97455 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.064s user 0.002s sys 0.010s
I20240502 14:14:56.239742 30556 raft_consensus.cc:348] T 9eb6af02e761412dae5ab888a4b97455 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:56.240103 30556 raft_consensus.cc:374] T 9eb6af02e761412dae5ab888a4b97455 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:56.240240 30556 raft_consensus.cc:729] T 9eb6af02e761412dae5ab888a4b97455 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:14:56.240670 30556 consensus_queue.cc:260] T 9eb6af02e761412dae5ab888a4b97455 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:56.248425 30556 ts_tablet_manager.cc:1398] T 9eb6af02e761412dae5ab888a4b97455 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 14:14:56.249544 30564 tablet_bootstrap.cc:492] T 82e51aa5371e4731bc00daafc65c7a88 P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:14:56.249792 30564 ts_tablet_manager.cc:1367] T 82e51aa5371e4731bc00daafc65c7a88 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.072s user 0.005s sys 0.004s
I20240502 14:14:56.250738 30564 raft_consensus.cc:348] T 82e51aa5371e4731bc00daafc65c7a88 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:56.251068 30564 raft_consensus.cc:374] T 82e51aa5371e4731bc00daafc65c7a88 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:56.251219 30564 raft_consensus.cc:729] T 82e51aa5371e4731bc00daafc65c7a88 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:14:56.251590 30564 consensus_queue.cc:260] T 82e51aa5371e4731bc00daafc65c7a88 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:56.251541 30470 raft_consensus.cc:2932] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:14:56.252149 26687 raft_consensus.cc:2932] T d7f02f897ce54fefb03832da3c2d95e0 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:14:56.266175 26818 raft_consensus.cc:2932] T d7f02f897ce54fefb03832da3c2d95e0 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:14:56.266723 30564 ts_tablet_manager.cc:1398] T 82e51aa5371e4731bc00daafc65c7a88 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20240502 14:14:56.268905 30568 tablet_copy_client.cc:783] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:56.269568 30568 tablet_copy_client.cc:647] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:56.269995 30601 tablet_copy_client.cc:308] T 2532ebd7bdf14876b74a26afd674a7ac P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.200:46097
I20240502 14:14:56.270998 29595 tablet_copy_service.cc:140] P 4092902b3e674857a0147f1b4f2cdb86: Received BeginTabletCopySession request for tablet 2532ebd7bdf14876b74a26afd674a7ac from peer 89af7f423cf44717a08d6b142f0ed49d ({username='slave'} at 127.19.94.193:43933)
I20240502 14:14:56.271188 29595 tablet_copy_service.cc:161] P 4092902b3e674857a0147f1b4f2cdb86: Beginning new tablet copy session on tablet 2532ebd7bdf14876b74a26afd674a7ac from peer 89af7f423cf44717a08d6b142f0ed49d at {username='slave'} at 127.19.94.193:43933: session id = 89af7f423cf44717a08d6b142f0ed49d-2532ebd7bdf14876b74a26afd674a7ac
I20240502 14:14:56.274133 29595 tablet_copy_source_session.cc:205] T 2532ebd7bdf14876b74a26afd674a7ac P 4092902b3e674857a0147f1b4f2cdb86: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:56.275182 30560 tablet_bootstrap.cc:492] T 9eb6af02e761412dae5ab888a4b97455 P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:14:56.275346 30560 ts_tablet_manager.cc:1367] T 9eb6af02e761412dae5ab888a4b97455 P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.095s user 0.000s sys 0.029s
I20240502 14:14:56.276186 30560 raft_consensus.cc:348] T 9eb6af02e761412dae5ab888a4b97455 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:56.276427 30560 raft_consensus.cc:374] T 9eb6af02e761412dae5ab888a4b97455 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:56.276525 30560 raft_consensus.cc:729] T 9eb6af02e761412dae5ab888a4b97455 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:14:56.276780 30560 consensus_queue.cc:260] T 9eb6af02e761412dae5ab888a4b97455 P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:56.277511 30560 ts_tablet_manager.cc:1398] T 9eb6af02e761412dae5ab888a4b97455 P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:56.277289 26279 catalog_manager.cc:5484] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a reported cstate change: config changed from index -1 to 4, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:14:56.278573 30606 tablet_bootstrap.cc:492] T 9eb6af02e761412dae5ab888a4b97455 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:56.278715 30606 ts_tablet_manager.cc:1367] T 9eb6af02e761412dae5ab888a4b97455 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.051s user 0.004s sys 0.000s
I20240502 14:14:56.279495 30606 raft_consensus.cc:348] T 9eb6af02e761412dae5ab888a4b97455 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:56.279788 30606 raft_consensus.cc:374] T 9eb6af02e761412dae5ab888a4b97455 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:56.279950 30606 raft_consensus.cc:729] T 9eb6af02e761412dae5ab888a4b97455 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:56.280261 30606 consensus_queue.cc:260] T 9eb6af02e761412dae5ab888a4b97455 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:56.281028 30606 ts_tablet_manager.cc:1398] T 9eb6af02e761412dae5ab888a4b97455 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:56.283440 30568 tablet_copy_client.cc:523] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:56.287746 30568 tablet_bootstrap.cc:492] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:56.288280 30601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2532ebd7bdf14876b74a26afd674a7ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:56.292421 30448 raft_consensus.cc:482] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:56.292588 30448 raft_consensus.cc:504] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:56.293334 30448 leader_election.cc:288] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:14:56.294193 29174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82e51aa5371e4731bc00daafc65c7a88" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:14:56.294450 29174 raft_consensus.cc:2454] T 82e51aa5371e4731bc00daafc65c7a88 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:14:56.295235 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82e51aa5371e4731bc00daafc65c7a88" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:14:56.295477 29443 raft_consensus.cc:2454] T 82e51aa5371e4731bc00daafc65c7a88 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:14:56.296170 29510 leader_election.cc:302] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 4092902b3e674857a0147f1b4f2cdb86; no voters:
I20240502 14:14:56.296718 30448 raft_consensus.cc:2781] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:56.296838 30448 raft_consensus.cc:482] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:56.296911 30448 raft_consensus.cc:3037] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:56.299651 30448 raft_consensus.cc:504] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:56.302027 29174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82e51aa5371e4731bc00daafc65c7a88" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:14:56.302246 29174 raft_consensus.cc:3037] T 82e51aa5371e4731bc00daafc65c7a88 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:56.306383 29174 raft_consensus.cc:2454] T 82e51aa5371e4731bc00daafc65c7a88 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:14:56.307711 29510 leader_election.cc:302] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 4092902b3e674857a0147f1b4f2cdb86; no voters:
I20240502 14:14:56.308071 30089 raft_consensus.cc:2781] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:56.309917 30089 raft_consensus.cc:686] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Becoming Leader. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Running, Role: LEADER
I20240502 14:14:56.310264 30089 consensus_queue.cc:237] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:56.312566 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82e51aa5371e4731bc00daafc65c7a88" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
I20240502 14:14:56.312803 29443 raft_consensus.cc:3037] T 82e51aa5371e4731bc00daafc65c7a88 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:56.315842 29443 raft_consensus.cc:2454] T 82e51aa5371e4731bc00daafc65c7a88 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:14:56.316262 30452 raft_consensus.cc:482] T 0d0298cfb57342f7b840a8656f452645 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:56.316443 30452 raft_consensus.cc:504] T 0d0298cfb57342f7b840a8656f452645 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:56.317193 30452 leader_election.cc:288] T 0d0298cfb57342f7b840a8656f452645 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:56.318233 29575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d0298cfb57342f7b840a8656f452645" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86" is_pre_election: true
I20240502 14:14:56.318594 29575 raft_consensus.cc:2379] T 0d0298cfb57342f7b840a8656f452645 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 68a3c2b17ed94e06aba66ed4178132ab in current term 1: Already voted for candidate e4cfb8fc5a474177af6848a6efa13a27 in this term.
I20240502 14:14:56.319885 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d0298cfb57342f7b840a8656f452645" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:14:56.320699 29379 leader_election.cc:302] T 0d0298cfb57342f7b840a8656f452645 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab; no voters: 4092902b3e674857a0147f1b4f2cdb86, e4cfb8fc5a474177af6848a6efa13a27
I20240502 14:14:56.321224 30452 raft_consensus.cc:3037] T 0d0298cfb57342f7b840a8656f452645 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:56.325091 30452 raft_consensus.cc:2726] T 0d0298cfb57342f7b840a8656f452645 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:14:56.325506 30448 leader_election.cc:288] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Requested vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:14:56.329821 26276 catalog_manager.cc:5484] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200). New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:14:56.340503 30601 tablet_copy_client.cc:783] T 2532ebd7bdf14876b74a26afd674a7ac P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:56.344096 30601 tablet_copy_client.cc:647] T 2532ebd7bdf14876b74a26afd674a7ac P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:56.346601 30601 tablet_copy_client.cc:523] T 2532ebd7bdf14876b74a26afd674a7ac P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:56.349229 30568 tablet_bootstrap.cc:492] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:56.349819 30568 tablet_bootstrap.cc:492] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap complete.
I20240502 14:14:56.350181 30568 ts_tablet_manager.cc:1367] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.063s user 0.018s sys 0.011s
I20240502 14:14:56.351153 30568 raft_consensus.cc:348] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:56.351483 30568 raft_consensus.cc:729] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:56.351840 30568 consensus_queue.cc:260] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:56.352658 30568 ts_tablet_manager.cc:1398] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:14:56.354282 29594 tablet_copy_service.cc:342] P 4092902b3e674857a0147f1b4f2cdb86: Request end of tablet copy session 69ab78b2a1b74d2bbf2cdeafa0a4af23-1704fbfbeed04dcaac8bc663b7ee2a5b received from {username='slave'} at 127.19.94.195:39709
I20240502 14:14:56.354457 29594 tablet_copy_service.cc:433] P 4092902b3e674857a0147f1b4f2cdb86: ending tablet copy session 69ab78b2a1b74d2bbf2cdeafa0a4af23-1704fbfbeed04dcaac8bc663b7ee2a5b on tablet 1704fbfbeed04dcaac8bc663b7ee2a5b with peer 69ab78b2a1b74d2bbf2cdeafa0a4af23
W20240502 14:14:56.357596 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f02f897ce54fefb03832da3c2d95e0. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:56.361876 30601 tablet_bootstrap.cc:492] T 2532ebd7bdf14876b74a26afd674a7ac P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:56.369560 30580 tablet_bootstrap.cc:492] T 310aa6ad3604487589dbe96afa25bb05 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:56.370239 30580 tablet_bootstrap.cc:492] T 310aa6ad3604487589dbe96afa25bb05 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap complete.
I20240502 14:14:56.370613 30580 ts_tablet_manager.cc:1367] T 310aa6ad3604487589dbe96afa25bb05 P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.283s user 0.058s sys 0.019s
I20240502 14:14:56.371122 30425 consensus_queue.cc:1035] T 0d0298cfb57342f7b840a8656f452645 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:56.371778 30580 raft_consensus.cc:348] T 310aa6ad3604487589dbe96afa25bb05 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:56.372177 30580 raft_consensus.cc:729] T 310aa6ad3604487589dbe96afa25bb05 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:14:56.372539 30580 consensus_queue.cc:260] T 310aa6ad3604487589dbe96afa25bb05 P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:56.373245 30580 ts_tablet_manager.cc:1398] T 310aa6ad3604487589dbe96afa25bb05 P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 14:14:56.374089 26578 tablet_copy_service.cc:342] P ba187e38a2754d60a12de9ac4856430a: Request end of tablet copy session 1cb67158133e4648a6eceba32d4dbba3-310aa6ad3604487589dbe96afa25bb05 received from {username='slave'} at 127.19.94.197:58769
I20240502 14:14:56.374228 26578 tablet_copy_service.cc:433] P ba187e38a2754d60a12de9ac4856430a: ending tablet copy session 1cb67158133e4648a6eceba32d4dbba3-310aa6ad3604487589dbe96afa25bb05 on tablet 310aa6ad3604487589dbe96afa25bb05 with peer 1cb67158133e4648a6eceba32d4dbba3
I20240502 14:14:56.384846 29791 consensus_queue.cc:1035] T 0d0298cfb57342f7b840a8656f452645 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:56.390169 30601 tablet_bootstrap.cc:492] T 2532ebd7bdf14876b74a26afd674a7ac P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:56.390787 30601 tablet_bootstrap.cc:492] T 2532ebd7bdf14876b74a26afd674a7ac P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap complete.
I20240502 14:14:56.391227 30601 ts_tablet_manager.cc:1367] T 2532ebd7bdf14876b74a26afd674a7ac P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.030s user 0.011s sys 0.000s
I20240502 14:14:56.393174 30509 tablet_bootstrap.cc:492] T 226b35583ee4441a82a66520aeffbb7d P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:56.395339 30509 tablet_bootstrap.cc:492] T 226b35583ee4441a82a66520aeffbb7d P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap complete.
I20240502 14:14:56.395478 30601 raft_consensus.cc:348] T 2532ebd7bdf14876b74a26afd674a7ac P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:56.395844 30601 raft_consensus.cc:729] T 2532ebd7bdf14876b74a26afd674a7ac P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:56.396296 30601 consensus_queue.cc:260] T 2532ebd7bdf14876b74a26afd674a7ac P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:56.397203 30601 ts_tablet_manager.cc:1398] T 2532ebd7bdf14876b74a26afd674a7ac P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:14:56.398631 29594 tablet_copy_service.cc:342] P 4092902b3e674857a0147f1b4f2cdb86: Request end of tablet copy session 89af7f423cf44717a08d6b142f0ed49d-2532ebd7bdf14876b74a26afd674a7ac received from {username='slave'} at 127.19.94.193:43933
I20240502 14:14:56.398763 29594 tablet_copy_service.cc:433] P 4092902b3e674857a0147f1b4f2cdb86: ending tablet copy session 89af7f423cf44717a08d6b142f0ed49d-2532ebd7bdf14876b74a26afd674a7ac on tablet 2532ebd7bdf14876b74a26afd674a7ac with peer 89af7f423cf44717a08d6b142f0ed49d
I20240502 14:14:56.396647 30509 ts_tablet_manager.cc:1367] T 226b35583ee4441a82a66520aeffbb7d P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.284s user 0.060s sys 0.000s
I20240502 14:14:56.404917 30509 raft_consensus.cc:348] T 226b35583ee4441a82a66520aeffbb7d P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:56.406453 30509 raft_consensus.cc:729] T 226b35583ee4441a82a66520aeffbb7d P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:14:56.407011 30509 consensus_queue.cc:260] T 226b35583ee4441a82a66520aeffbb7d P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:56.408675 30509 ts_tablet_manager.cc:1398] T 226b35583ee4441a82a66520aeffbb7d P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20240502 14:14:56.409454 26447 tablet_copy_service.cc:342] P 89af7f423cf44717a08d6b142f0ed49d: Request end of tablet copy session 4092902b3e674857a0147f1b4f2cdb86-226b35583ee4441a82a66520aeffbb7d received from {username='slave'} at 127.19.94.200:60351
I20240502 14:14:56.409570 26447 tablet_copy_service.cc:433] P 89af7f423cf44717a08d6b142f0ed49d: ending tablet copy session 4092902b3e674857a0147f1b4f2cdb86-226b35583ee4441a82a66520aeffbb7d on tablet 226b35583ee4441a82a66520aeffbb7d with peer 4092902b3e674857a0147f1b4f2cdb86
I20240502 14:14:56.416347 30448 raft_consensus.cc:482] T 9eb6af02e761412dae5ab888a4b97455 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:56.416548 30448 raft_consensus.cc:504] T 9eb6af02e761412dae5ab888a4b97455 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:56.417145 30448 leader_election.cc:288] T 9eb6af02e761412dae5ab888a4b97455 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:14:56.417771 26423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb6af02e761412dae5ab888a4b97455" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:14:56.417745 29308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb6af02e761412dae5ab888a4b97455" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:14:56.417999 26423 raft_consensus.cc:2454] T 9eb6af02e761412dae5ab888a4b97455 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:14:56.418030 29308 raft_consensus.cc:2454] T 9eb6af02e761412dae5ab888a4b97455 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:14:56.418715 29510 leader_election.cc:302] T 9eb6af02e761412dae5ab888a4b97455 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:14:56.419118 30448 raft_consensus.cc:2781] T 9eb6af02e761412dae5ab888a4b97455 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:56.419245 30448 raft_consensus.cc:482] T 9eb6af02e761412dae5ab888a4b97455 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:56.419343 30448 raft_consensus.cc:3037] T 9eb6af02e761412dae5ab888a4b97455 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:56.420276 26687 raft_consensus.cc:3037] T f4973d1d77904cb9a3cf8d49346b330e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:56.421485 30448 raft_consensus.cc:504] T 9eb6af02e761412dae5ab888a4b97455 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:56.422036 30448 leader_election.cc:288] T 9eb6af02e761412dae5ab888a4b97455 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Requested vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:14:56.422495 26423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb6af02e761412dae5ab888a4b97455" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:14:56.422700 26423 raft_consensus.cc:3037] T 9eb6af02e761412dae5ab888a4b97455 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:56.422693 29308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb6af02e761412dae5ab888a4b97455" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
I20240502 14:14:56.422930 29308 raft_consensus.cc:3037] T 9eb6af02e761412dae5ab888a4b97455 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:56.425333 29308 raft_consensus.cc:2454] T 9eb6af02e761412dae5ab888a4b97455 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:14:56.425748 29510 leader_election.cc:302] T 9eb6af02e761412dae5ab888a4b97455 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:14:56.426306 30448 raft_consensus.cc:2781] T 9eb6af02e761412dae5ab888a4b97455 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:56.426265 30515 consensus_queue.cc:1035] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:56.426885 30448 raft_consensus.cc:686] T 9eb6af02e761412dae5ab888a4b97455 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Becoming Leader. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Running, Role: LEADER
I20240502 14:14:56.427201 30448 consensus_queue.cc:237] T 9eb6af02e761412dae5ab888a4b97455 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:56.425778 26423 raft_consensus.cc:2454] T 9eb6af02e761412dae5ab888a4b97455 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:14:56.431882 26285 catalog_manager.cc:5484] T 9eb6af02e761412dae5ab888a4b97455 P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200). New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:14:56.439633 30274 consensus_queue.cc:1035] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:56.460254 26557 consensus_queue.cc:237] T d1aa4f4be5894ec5a64275f1172e1f55 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:56.466234 26818 raft_consensus.cc:1261] T d1aa4f4be5894ec5a64275f1172e1f55 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:14:56.466548 26493 consensus_peers.cc:485] T d1aa4f4be5894ec5a64275f1172e1f55 P ba187e38a2754d60a12de9ac4856430a -> Peer 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): Couldn't send request to peer 1cb67158133e4648a6eceba32d4dbba3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1aa4f4be5894ec5a64275f1172e1f55. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:56.466843 26423 raft_consensus.cc:1261] T d1aa4f4be5894ec5a64275f1172e1f55 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:56.466828 30615 consensus_queue.cc:1035] T d1aa4f4be5894ec5a64275f1172e1f55 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:56.467435 30615 consensus_queue.cc:1035] T d1aa4f4be5894ec5a64275f1172e1f55 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:56.478591 30470 raft_consensus.cc:2932] T d1aa4f4be5894ec5a64275f1172e1f55 P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:14:56.479264 26819 raft_consensus.cc:2932] T d1aa4f4be5894ec5a64275f1172e1f55 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:14:56.482896 26423 raft_consensus.cc:2932] T d1aa4f4be5894ec5a64275f1172e1f55 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:14:56.484797 26285 catalog_manager.cc:5484] T d1aa4f4be5894ec5a64275f1172e1f55 P ba187e38a2754d60a12de9ac4856430a reported cstate change: config changed from index -1 to 4, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:14:56.512754 29176 raft_consensus.cc:1203] T 310aa6ad3604487589dbe96afa25bb05 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:56.524847 26557 consensus_queue.cc:237] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:56.527371 26687 raft_consensus.cc:1261] T c586354e10cb4bedbbb20bd103ad477e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:14:56.527916 26494 consensus_peers.cc:485] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a -> Peer 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097): Couldn't send request to peer 4092902b3e674857a0147f1b4f2cdb86. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c586354e10cb4bedbbb20bd103ad477e. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:56.528343 30615 consensus_queue.cc:1035] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:56.528633 26819 raft_consensus.cc:1261] T c586354e10cb4bedbbb20bd103ad477e P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:56.529268 30615 consensus_queue.cc:1035] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:56.532003 30615 raft_consensus.cc:2932] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:14:56.532682 26819 raft_consensus.cc:2932] T c586354e10cb4bedbbb20bd103ad477e P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:14:56.539404 26285 catalog_manager.cc:5484] T c586354e10cb4bedbbb20bd103ad477e P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: config changed from index -1 to 4, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:14:56.539426 26687 raft_consensus.cc:2932] T c586354e10cb4bedbbb20bd103ad477e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:14:56.552685 26276 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:14:56.561167 29308 raft_consensus.cc:1261] T 9eb6af02e761412dae5ab888a4b97455 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:56.562328 26423 raft_consensus.cc:1261] T 9eb6af02e761412dae5ab888a4b97455 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:56.562357 30089 consensus_queue.cc:1035] T 9eb6af02e761412dae5ab888a4b97455 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:56.563755 30089 consensus_queue.cc:1035] T 9eb6af02e761412dae5ab888a4b97455 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:56.570253 26276 catalog_manager.cc:5576] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659) does not have the latest schema for tablet 9eb6af02e761412dae5ab888a4b97455 (table rebalancer_dynamic_table [id=ec92b676b3e04dcc8943814a451c0c4a]). Expected version 1 got 0
I20240502 14:14:56.572032 29575 raft_consensus.cc:1203] T 226b35583ee4441a82a66520aeffbb7d P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:56.577423 30677 tablet.cc:1722] T 9eb6af02e761412dae5ab888a4b97455 P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:56.578099 30678 tablet.cc:1722] T 9eb6af02e761412dae5ab888a4b97455 P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:56.579365 30678 tablet.cc:1620] T 9eb6af02e761412dae5ab888a4b97455 P c1100e16b3b9495ca61d4f95e89b6a89: MemRowSet was empty: no flush needed.
I20240502 14:14:56.579092 30677 tablet.cc:1620] T 9eb6af02e761412dae5ab888a4b97455 P 4092902b3e674857a0147f1b4f2cdb86: MemRowSet was empty: no flush needed.
I20240502 14:14:56.581099 26276 catalog_manager.cc:5576] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659) does not have the latest schema for tablet 9eb6af02e761412dae5ab888a4b97455 (table rebalancer_dynamic_table [id=ec92b676b3e04dcc8943814a451c0c4a]). Expected version 1 got 0
I20240502 14:14:56.595376 26276 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=ec92b676b3e04dcc8943814a451c0c4a] alter complete (version 1)
I20240502 14:14:56.599205 26276 catalog_manager.cc:5576] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985) does not have the latest schema for tablet 9eb6af02e761412dae5ab888a4b97455 (table rebalancer_dynamic_table [id=ec92b676b3e04dcc8943814a451c0c4a]). Expected version 1 got 0
I20240502 14:14:56.600059 30680 tablet.cc:1722] T 9eb6af02e761412dae5ab888a4b97455 P 89af7f423cf44717a08d6b142f0ed49d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:56.602653 30680 tablet.cc:1620] T 9eb6af02e761412dae5ab888a4b97455 P 89af7f423cf44717a08d6b142f0ed49d: MemRowSet was empty: no flush needed.
I20240502 14:14:56.603799 26276 catalog_manager.cc:5576] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985) does not have the latest schema for tablet 9eb6af02e761412dae5ab888a4b97455 (table rebalancer_dynamic_table [id=ec92b676b3e04dcc8943814a451c0c4a]). Expected version 1 got 0
I20240502 14:14:56.639452 26819 consensus_queue.cc:237] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:56.642026 26689 raft_consensus.cc:1261] T 533b42de2de14abc96c00a928d5deed1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:14:56.642295 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 533b42de2de14abc96c00a928d5deed1. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:56.642302 26423 raft_consensus.cc:1261] T 533b42de2de14abc96c00a928d5deed1 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:56.642818 30425 consensus_queue.cc:1035] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:56.643095 29791 consensus_queue.cc:1035] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:56.646535 30425 raft_consensus.cc:2932] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:14:56.649083 26423 raft_consensus.cc:2932] T 533b42de2de14abc96c00a928d5deed1 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:14:56.651919 26689 raft_consensus.cc:2932] T 533b42de2de14abc96c00a928d5deed1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:14:56.652355 26276 catalog_manager.cc:5484] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: config changed from index -1 to 4, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:14:56.691052 26423 raft_consensus.cc:1203] T 2532ebd7bdf14876b74a26afd674a7ac P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:14:56.694247 26557 consensus_queue.cc:237] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:56.696537 26689 raft_consensus.cc:1261] T 66554eb8b44145d4b474707b061c9659 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:56.696641 26423 raft_consensus.cc:1261] T 66554eb8b44145d4b474707b061c9659 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:14:56.696951 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 66554eb8b44145d4b474707b061c9659. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:56.697163 30470 consensus_queue.cc:1035] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:56.697440 30615 consensus_queue.cc:1035] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:56.697515 26285 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:14:56.701675 30470 raft_consensus.cc:2932] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:14:56.706169 26689 raft_consensus.cc:2932] T 66554eb8b44145d4b474707b061c9659 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:14:56.714121 26687 raft_consensus.cc:1203] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:14:56.717049 30677 tablet.cc:1722] T 9eb6af02e761412dae5ab888a4b97455 P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:56.717604 30677 tablet.cc:1620] T 9eb6af02e761412dae5ab888a4b97455 P 4092902b3e674857a0147f1b4f2cdb86: MemRowSet was empty: no flush needed.
I20240502 14:14:56.718027 26276 catalog_manager.cc:5484] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a reported cstate change: config changed from index -1 to 4, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:14:56.721295 30713 ts_tablet_manager.cc:902] T d7f02f897ce54fefb03832da3c2d95e0 P c1100e16b3b9495ca61d4f95e89b6a89: Initiating tablet copy from peer ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:14:56.730073 30678 tablet.cc:1722] T 9eb6af02e761412dae5ab888a4b97455 P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:56.730659 30678 tablet.cc:1620] T 9eb6af02e761412dae5ab888a4b97455 P c1100e16b3b9495ca61d4f95e89b6a89: MemRowSet was empty: no flush needed.
I20240502 14:14:56.735525 30680 tablet.cc:1722] T 9eb6af02e761412dae5ab888a4b97455 P 89af7f423cf44717a08d6b142f0ed49d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:56.736093 30680 tablet.cc:1620] T 9eb6af02e761412dae5ab888a4b97455 P 89af7f423cf44717a08d6b142f0ed49d: MemRowSet was empty: no flush needed.
I20240502 14:14:56.740964 26423 raft_consensus.cc:2932] T 66554eb8b44145d4b474707b061c9659 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:14:56.744149 26276 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=ec92b676b3e04dcc8943814a451c0c4a] alter complete (version 2)
I20240502 14:14:56.758780 30713 tablet_copy_client.cc:308] T d7f02f897ce54fefb03832da3c2d95e0 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.194:36025
I20240502 14:14:56.759641 26578 tablet_copy_service.cc:140] P ba187e38a2754d60a12de9ac4856430a: Received BeginTabletCopySession request for tablet d7f02f897ce54fefb03832da3c2d95e0 from peer c1100e16b3b9495ca61d4f95e89b6a89 ({username='slave'} at 127.19.94.198:36021)
I20240502 14:14:56.759806 26578 tablet_copy_service.cc:161] P ba187e38a2754d60a12de9ac4856430a: Beginning new tablet copy session on tablet d7f02f897ce54fefb03832da3c2d95e0 from peer c1100e16b3b9495ca61d4f95e89b6a89 at {username='slave'} at 127.19.94.198:36021: session id = c1100e16b3b9495ca61d4f95e89b6a89-d7f02f897ce54fefb03832da3c2d95e0
I20240502 14:14:56.762779 26578 tablet_copy_source_session.cc:205] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:56.764163 30713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7f02f897ce54fefb03832da3c2d95e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:56.769600 30713 tablet_copy_client.cc:783] T d7f02f897ce54fefb03832da3c2d95e0 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:56.770013 30713 tablet_copy_client.cc:647] T d7f02f897ce54fefb03832da3c2d95e0 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:56.771929 30713 tablet_copy_client.cc:523] T d7f02f897ce54fefb03832da3c2d95e0 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:56.775429 30713 tablet_bootstrap.cc:492] T d7f02f897ce54fefb03832da3c2d95e0 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:14:56.795934 26557 consensus_queue.cc:237] T 62218b2a597e4fb188ced6d2b9a83a59 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
W20240502 14:14:56.802363 26494 consensus_peers.cc:485] T 62218b2a597e4fb188ced6d2b9a83a59 P ba187e38a2754d60a12de9ac4856430a -> Peer 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097): Couldn't send request to peer 4092902b3e674857a0147f1b4f2cdb86. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62218b2a597e4fb188ced6d2b9a83a59. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:56.803315 26819 raft_consensus.cc:1261] T 62218b2a597e4fb188ced6d2b9a83a59 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:56.803364 26689 raft_consensus.cc:1261] T 62218b2a597e4fb188ced6d2b9a83a59 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:56.803866 30470 consensus_queue.cc:1035] T 62218b2a597e4fb188ced6d2b9a83a59 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:56.804217 30615 consensus_queue.cc:1035] T 62218b2a597e4fb188ced6d2b9a83a59 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:56.807940 30470 raft_consensus.cc:2932] T 62218b2a597e4fb188ced6d2b9a83a59 P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:14:56.808616 26689 raft_consensus.cc:2932] T 62218b2a597e4fb188ced6d2b9a83a59 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:14:56.813840 26819 raft_consensus.cc:2932] T 62218b2a597e4fb188ced6d2b9a83a59 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:14:56.814711 26276 catalog_manager.cc:5484] T 62218b2a597e4fb188ced6d2b9a83a59 P ba187e38a2754d60a12de9ac4856430a reported cstate change: config changed from index -1 to 4, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:14:56.829587 30713 tablet_bootstrap.cc:492] T d7f02f897ce54fefb03832da3c2d95e0 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:56.830004 30713 tablet_bootstrap.cc:492] T d7f02f897ce54fefb03832da3c2d95e0 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap complete.
I20240502 14:14:56.830293 30713 ts_tablet_manager.cc:1367] T d7f02f897ce54fefb03832da3c2d95e0 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.055s user 0.048s sys 0.002s
I20240502 14:14:56.830978 30713 raft_consensus.cc:348] T d7f02f897ce54fefb03832da3c2d95e0 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:56.831205 30713 raft_consensus.cc:729] T d7f02f897ce54fefb03832da3c2d95e0 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:14:56.831494 30713 consensus_queue.cc:260] T d7f02f897ce54fefb03832da3c2d95e0 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:56.832309 30713 ts_tablet_manager.cc:1398] T d7f02f897ce54fefb03832da3c2d95e0 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 14:14:56.833509 26578 tablet_copy_service.cc:342] P ba187e38a2754d60a12de9ac4856430a: Request end of tablet copy session c1100e16b3b9495ca61d4f95e89b6a89-d7f02f897ce54fefb03832da3c2d95e0 received from {username='slave'} at 127.19.94.198:36021
I20240502 14:14:56.833693 26578 tablet_copy_service.cc:433] P ba187e38a2754d60a12de9ac4856430a: ending tablet copy session c1100e16b3b9495ca61d4f95e89b6a89-d7f02f897ce54fefb03832da3c2d95e0 on tablet d7f02f897ce54fefb03832da3c2d95e0 with peer c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:14:56.835183 30089 consensus_queue.cc:1035] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:56.846236 30448 consensus_queue.cc:1035] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:56.853348 26285 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:14:56.862689 30677 tablet.cc:1722] T 9eb6af02e761412dae5ab888a4b97455 P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:56.864445 30678 tablet.cc:1722] T 9eb6af02e761412dae5ab888a4b97455 P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:56.868957 30680 tablet.cc:1722] T 9eb6af02e761412dae5ab888a4b97455 P 89af7f423cf44717a08d6b142f0ed49d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:56.874543 26285 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table_new_name [id=ec92b676b3e04dcc8943814a451c0c4a] alter complete (version 3)
I20240502 14:14:56.886211 29575 consensus_queue.cc:237] T f8923ed1ce94441a8eb303b6f7dd3acb P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:56.888772 30756 ts_tablet_manager.cc:902] T d1aa4f4be5894ec5a64275f1172e1f55 P 1cb67158133e4648a6eceba32d4dbba3: Initiating tablet copy from peer ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:14:56.889683 26689 raft_consensus.cc:1261] T f8923ed1ce94441a8eb303b6f7dd3acb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:14:56.889921 29510 consensus_peers.cc:485] T f8923ed1ce94441a8eb303b6f7dd3acb P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): Couldn't send request to peer 1cb67158133e4648a6eceba32d4dbba3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f8923ed1ce94441a8eb303b6f7dd3acb. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:56.890048 29443 raft_consensus.cc:1261] T f8923ed1ce94441a8eb303b6f7dd3acb P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:56.890409 30448 consensus_queue.cc:1035] T f8923ed1ce94441a8eb303b6f7dd3acb P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:14:56.891547 30448 consensus_queue.cc:1035] T f8923ed1ce94441a8eb303b6f7dd3acb P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:14:56.892315 30756 tablet_copy_client.cc:308] T d1aa4f4be5894ec5a64275f1172e1f55 P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.194:36025
I20240502 14:14:56.893316 26578 tablet_copy_service.cc:140] P ba187e38a2754d60a12de9ac4856430a: Received BeginTabletCopySession request for tablet d1aa4f4be5894ec5a64275f1172e1f55 from peer 1cb67158133e4648a6eceba32d4dbba3 ({username='slave'} at 127.19.94.197:58769)
I20240502 14:14:56.893505 26578 tablet_copy_service.cc:161] P ba187e38a2754d60a12de9ac4856430a: Beginning new tablet copy session on tablet d1aa4f4be5894ec5a64275f1172e1f55 from peer 1cb67158133e4648a6eceba32d4dbba3 at {username='slave'} at 127.19.94.197:58769: session id = 1cb67158133e4648a6eceba32d4dbba3-d1aa4f4be5894ec5a64275f1172e1f55
I20240502 14:14:56.895694 26578 tablet_copy_source_session.cc:205] T d1aa4f4be5894ec5a64275f1172e1f55 P ba187e38a2754d60a12de9ac4856430a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:56.897048 30756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1aa4f4be5894ec5a64275f1172e1f55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:56.897665 30619 raft_consensus.cc:2932] T f8923ed1ce94441a8eb303b6f7dd3acb P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:14:56.898150 26689 raft_consensus.cc:2932] T f8923ed1ce94441a8eb303b6f7dd3acb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:14:56.903928 26276 catalog_manager.cc:5484] T f8923ed1ce94441a8eb303b6f7dd3acb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: config changed from index -1 to 2, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:14:56.904976 29443 raft_consensus.cc:2932] T f8923ed1ce94441a8eb303b6f7dd3acb P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:14:56.907018 30756 tablet_copy_client.cc:783] T d1aa4f4be5894ec5a64275f1172e1f55 P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:56.907531 30756 tablet_copy_client.cc:647] T d1aa4f4be5894ec5a64275f1172e1f55 P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:56.916877 30756 tablet_copy_client.cc:523] T d1aa4f4be5894ec5a64275f1172e1f55 P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:56.920459 30756 tablet_bootstrap.cc:492] T d1aa4f4be5894ec5a64275f1172e1f55 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:14:56.939184 26276 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_extra_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:56.950385 29154 tablet_service.cc:1467] Processing CreateTablet for tablet a262ac98859b4208b70573662cb898cb (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=15e0d2ef8fa244be97447fb977d23693]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:56.950682 29554 tablet_service.cc:1467] Processing CreateTablet for tablet a262ac98859b4208b70573662cb898cb (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=15e0d2ef8fa244be97447fb977d23693]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:56.950914 29154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a262ac98859b4208b70573662cb898cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:56.951143 29554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a262ac98859b4208b70573662cb898cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:56.952562 29288 tablet_service.cc:1467] Processing CreateTablet for tablet 822e90c09531419e9b32af0438fceb57 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=15e0d2ef8fa244be97447fb977d23693]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:56.952988 29288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 822e90c09531419e9b32af0438fceb57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:56.955765 29287 tablet_service.cc:1467] Processing CreateTablet for tablet 2434830a866e4f7dbe7c8fa18d48e735 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=15e0d2ef8fa244be97447fb977d23693]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:56.956184 29287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2434830a866e4f7dbe7c8fa18d48e735. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:56.958498 29422 tablet_service.cc:1467] Processing CreateTablet for tablet a262ac98859b4208b70573662cb898cb (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=15e0d2ef8fa244be97447fb977d23693]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:56.958984 29422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a262ac98859b4208b70573662cb898cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:56.960254 29156 tablet_service.cc:1467] Processing CreateTablet for tablet 2434830a866e4f7dbe7c8fa18d48e735 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=15e0d2ef8fa244be97447fb977d23693]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:56.960654 29156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2434830a866e4f7dbe7c8fa18d48e735. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:56.961341 29553 tablet_service.cc:1467] Processing CreateTablet for tablet 2434830a866e4f7dbe7c8fa18d48e735 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=15e0d2ef8fa244be97447fb977d23693]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:56.961777 29553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2434830a866e4f7dbe7c8fa18d48e735. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:56.966208 29423 tablet_service.cc:1467] Processing CreateTablet for tablet 822e90c09531419e9b32af0438fceb57 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=15e0d2ef8fa244be97447fb977d23693]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:56.966228 26406 tablet_service.cc:1467] Processing CreateTablet for tablet 822e90c09531419e9b32af0438fceb57 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=15e0d2ef8fa244be97447fb977d23693]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:56.966670 26406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 822e90c09531419e9b32af0438fceb57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:56.966674 29423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 822e90c09531419e9b32af0438fceb57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:56.974293 30777 tablet_bootstrap.cc:492] T a262ac98859b4208b70573662cb898cb P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:14:56.980331 30779 tablet_bootstrap.cc:492] T 2434830a866e4f7dbe7c8fa18d48e735 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:14:56.982354 30779 tablet_bootstrap.cc:654] T 2434830a866e4f7dbe7c8fa18d48e735 P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:56.984650 26276 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:14:56.984937 26276 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:14:56.993168 30780 tablet_bootstrap.cc:492] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:56.995548 26276 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet 9eb6af02e761412dae5ab888a4b97455
I20240502 14:14:56.997543 26406 tablet_service.cc:1514] Processing DeleteTablet for tablet 9eb6af02e761412dae5ab888a4b97455 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:56 UTC) from {username='slave'} at 127.0.0.1:37422
I20240502 14:14:56.998091 30777 tablet_bootstrap.cc:654] T a262ac98859b4208b70573662cb898cb P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:56.998344 29287 tablet_service.cc:1514] Processing DeleteTablet for tablet 9eb6af02e761412dae5ab888a4b97455 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:56 UTC) from {username='slave'} at 127.0.0.1:59160
I20240502 14:14:56.998414 29553 tablet_service.cc:1514] Processing DeleteTablet for tablet 9eb6af02e761412dae5ab888a4b97455 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:56 UTC) from {username='slave'} at 127.0.0.1:36398
I20240502 14:14:56.999756 30776 tablet_bootstrap.cc:492] T a262ac98859b4208b70573662cb898cb P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:14:57.001904 30776 tablet_bootstrap.cc:654] T a262ac98859b4208b70573662cb898cb P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:57.002099 30778 tablet_bootstrap.cc:492] T a262ac98859b4208b70573662cb898cb P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:14:57.003448 30780 tablet_bootstrap.cc:654] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:57.004031 30778 tablet_bootstrap.cc:654] T a262ac98859b4208b70573662cb898cb P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:57.008556 30778 tablet_bootstrap.cc:492] T a262ac98859b4208b70573662cb898cb P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:14:57.008827 30778 ts_tablet_manager.cc:1367] T a262ac98859b4208b70573662cb898cb P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 14:14:57.008884 30782 tablet_replica.cc:330] T 9eb6af02e761412dae5ab888a4b97455 P 89af7f423cf44717a08d6b142f0ed49d: stopping tablet replica
I20240502 14:14:57.009241 30782 raft_consensus.cc:2229] T 9eb6af02e761412dae5ab888a4b97455 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:57.009647 30782 raft_consensus.cc:2258] T 9eb6af02e761412dae5ab888a4b97455 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:57.010891 30782 ts_tablet_manager.cc:1875] T 9eb6af02e761412dae5ab888a4b97455 P 89af7f423cf44717a08d6b142f0ed49d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:57.012050 30778 raft_consensus.cc:348] T a262ac98859b4208b70573662cb898cb P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:57.012432 30778 raft_consensus.cc:374] T a262ac98859b4208b70573662cb898cb P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:57.012594 30778 raft_consensus.cc:729] T a262ac98859b4208b70573662cb898cb P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:14:57.012918 30778 consensus_queue.cc:260] T a262ac98859b4208b70573662cb898cb P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:57.016893 30778 ts_tablet_manager.cc:1398] T a262ac98859b4208b70573662cb898cb P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 14:14:57.017369 30784 tablet_replica.cc:330] T 9eb6af02e761412dae5ab888a4b97455 P 4092902b3e674857a0147f1b4f2cdb86: stopping tablet replica
I20240502 14:14:57.017972 30784 raft_consensus.cc:2229] T 9eb6af02e761412dae5ab888a4b97455 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:57.018442 30784 raft_consensus.cc:2258] T 9eb6af02e761412dae5ab888a4b97455 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:57.019131 30778 tablet_bootstrap.cc:492] T 822e90c09531419e9b32af0438fceb57 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:14:57.019695 30784 ts_tablet_manager.cc:1875] T 9eb6af02e761412dae5ab888a4b97455 P 4092902b3e674857a0147f1b4f2cdb86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:57.021714 30777 tablet_bootstrap.cc:492] T a262ac98859b4208b70573662cb898cb P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:14:57.021955 30777 ts_tablet_manager.cc:1367] T a262ac98859b4208b70573662cb898cb P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.048s user 0.004s sys 0.000s
I20240502 14:14:57.023962 30777 raft_consensus.cc:348] T a262ac98859b4208b70573662cb898cb P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:57.024255 30777 raft_consensus.cc:374] T a262ac98859b4208b70573662cb898cb P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:57.024384 30777 raft_consensus.cc:729] T a262ac98859b4208b70573662cb898cb P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:14:57.024753 30777 consensus_queue.cc:260] T a262ac98859b4208b70573662cb898cb P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:57.026582 30777 ts_tablet_manager.cc:1398] T a262ac98859b4208b70573662cb898cb P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:57.026913 30777 tablet_bootstrap.cc:492] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:14:57.027101 30779 tablet_bootstrap.cc:492] T 2434830a866e4f7dbe7c8fa18d48e735 P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:14:57.027330 30779 ts_tablet_manager.cc:1367] T 2434830a866e4f7dbe7c8fa18d48e735 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.009s
I20240502 14:14:57.029791 30779 raft_consensus.cc:348] T 2434830a866e4f7dbe7c8fa18d48e735 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:57.030113 30779 raft_consensus.cc:374] T 2434830a866e4f7dbe7c8fa18d48e735 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:57.030261 30779 raft_consensus.cc:729] T 2434830a866e4f7dbe7c8fa18d48e735 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:14:57.030423 30776 tablet_bootstrap.cc:492] T a262ac98859b4208b70573662cb898cb P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:14:57.030637 30776 ts_tablet_manager.cc:1367] T a262ac98859b4208b70573662cb898cb P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.000s
I20240502 14:14:57.030608 30779 consensus_queue.cc:260] T 2434830a866e4f7dbe7c8fa18d48e735 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:57.031514 30776 raft_consensus.cc:348] T a262ac98859b4208b70573662cb898cb P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:57.032305 30779 ts_tablet_manager.cc:1398] T 2434830a866e4f7dbe7c8fa18d48e735 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240502 14:14:57.032681 30779 tablet_bootstrap.cc:492] T 822e90c09531419e9b32af0438fceb57 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:14:57.033221 30777 tablet_bootstrap.cc:654] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:57.035302 30780 tablet_bootstrap.cc:492] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:57.035533 30780 ts_tablet_manager.cc:1367] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.042s user 0.002s sys 0.002s
I20240502 14:14:57.036437 30780 raft_consensus.cc:348] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:57.036720 30780 raft_consensus.cc:374] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:57.036875 30780 raft_consensus.cc:729] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:57.037181 30780 consensus_queue.cc:260] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:57.037823 30780 ts_tablet_manager.cc:1398] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:14:57.039031 30776 raft_consensus.cc:374] T a262ac98859b4208b70573662cb898cb P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:57.039237 30779 tablet_bootstrap.cc:654] T 822e90c09531419e9b32af0438fceb57 P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:57.040704 30776 raft_consensus.cc:729] T a262ac98859b4208b70573662cb898cb P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:14:57.041069 30776 consensus_queue.cc:260] T a262ac98859b4208b70573662cb898cb P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:57.042852 30776 ts_tablet_manager.cc:1398] T a262ac98859b4208b70573662cb898cb P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240502 14:14:57.043035 30784 ts_tablet_manager.cc:1888] T 9eb6af02e761412dae5ab888a4b97455 P 4092902b3e674857a0147f1b4f2cdb86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:57.043217 30784 log.cc:1196] T 9eb6af02e761412dae5ab888a4b97455 P 4092902b3e674857a0147f1b4f2cdb86: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/wal/wals/9eb6af02e761412dae5ab888a4b97455
I20240502 14:14:57.043254 30776 tablet_bootstrap.cc:492] T 2434830a866e4f7dbe7c8fa18d48e735 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:14:57.045662 30776 tablet_bootstrap.cc:654] T 2434830a866e4f7dbe7c8fa18d48e735 P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:57.045993 30784 ts_tablet_manager.cc:1909] T 9eb6af02e761412dae5ab888a4b97455 P 4092902b3e674857a0147f1b4f2cdb86: Deleting consensus metadata
I20240502 14:14:57.048543 30783 tablet_replica.cc:330] T 9eb6af02e761412dae5ab888a4b97455 P c1100e16b3b9495ca61d4f95e89b6a89: stopping tablet replica
I20240502 14:14:57.048893 30783 raft_consensus.cc:2229] T 9eb6af02e761412dae5ab888a4b97455 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:57.049260 30783 raft_consensus.cc:2258] T 9eb6af02e761412dae5ab888a4b97455 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:57.049729 30776 tablet_bootstrap.cc:492] T 2434830a866e4f7dbe7c8fa18d48e735 P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:14:57.050524 30776 ts_tablet_manager.cc:1367] T 2434830a866e4f7dbe7c8fa18d48e735 P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20240502 14:14:57.050854 30783 ts_tablet_manager.cc:1875] T 9eb6af02e761412dae5ab888a4b97455 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:57.052028 30776 raft_consensus.cc:348] T 2434830a866e4f7dbe7c8fa18d48e735 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:57.053179 30776 raft_consensus.cc:374] T 2434830a866e4f7dbe7c8fa18d48e735 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:57.053633 30776 raft_consensus.cc:729] T 2434830a866e4f7dbe7c8fa18d48e735 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:14:57.054036 30776 consensus_queue.cc:260] T 2434830a866e4f7dbe7c8fa18d48e735 P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:57.055016 30779 tablet_bootstrap.cc:492] T 822e90c09531419e9b32af0438fceb57 P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:14:57.055236 30779 ts_tablet_manager.cc:1367] T 822e90c09531419e9b32af0438fceb57 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.003s
I20240502 14:14:57.056140 30779 raft_consensus.cc:348] T 822e90c09531419e9b32af0438fceb57 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:57.056465 30779 raft_consensus.cc:374] T 822e90c09531419e9b32af0438fceb57 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:57.056612 30779 raft_consensus.cc:729] T 822e90c09531419e9b32af0438fceb57 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:14:57.056955 30779 consensus_queue.cc:260] T 822e90c09531419e9b32af0438fceb57 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:57.058743 30779 ts_tablet_manager.cc:1398] T 822e90c09531419e9b32af0438fceb57 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:57.054211 30619 raft_consensus.cc:482] T a262ac98859b4208b70573662cb898cb P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:57.059109 30619 raft_consensus.cc:504] T a262ac98859b4208b70573662cb898cb P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:57.059854 30619 leader_election.cc:288] T a262ac98859b4208b70573662cb898cb P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:14:57.060918 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a262ac98859b4208b70573662cb898cb" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:14:57.061182 29443 raft_consensus.cc:2454] T a262ac98859b4208b70573662cb898cb P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:14:57.061980 29176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a262ac98859b4208b70573662cb898cb" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:14:57.062244 29176 raft_consensus.cc:2454] T a262ac98859b4208b70573662cb898cb P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:14:57.063357 30778 tablet_bootstrap.cc:654] T 822e90c09531419e9b32af0438fceb57 P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:57.064226 26272 catalog_manager.cc:4830] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097): tablet 9eb6af02e761412dae5ab888a4b97455 (table rebalancer_dynamic_table_new_name [id=ec92b676b3e04dcc8943814a451c0c4a]) successfully deleted
I20240502 14:14:57.064982 29510 leader_election.cc:302] T a262ac98859b4208b70573662cb898cb P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 4092902b3e674857a0147f1b4f2cdb86; no voters:
I20240502 14:14:57.065234 30777 tablet_bootstrap.cc:492] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:14:57.065521 30777 ts_tablet_manager.cc:1367] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.039s user 0.000s sys 0.004s
I20240502 14:14:57.067184 30619 raft_consensus.cc:2781] T a262ac98859b4208b70573662cb898cb P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:57.067297 30619 raft_consensus.cc:482] T a262ac98859b4208b70573662cb898cb P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:57.067420 30619 raft_consensus.cc:3037] T a262ac98859b4208b70573662cb898cb P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:57.068490 30777 raft_consensus.cc:348] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:57.068895 30777 raft_consensus.cc:374] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:57.069037 30777 raft_consensus.cc:729] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:14:57.069376 30777 consensus_queue.cc:260] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:57.071161 30619 raft_consensus.cc:504] T a262ac98859b4208b70573662cb898cb P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:57.072978 29176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a262ac98859b4208b70573662cb898cb" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
W20240502 14:14:57.072747 26363 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 46463.8us
I20240502 14:14:57.073212 29176 raft_consensus.cc:3037] T a262ac98859b4208b70573662cb898cb P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:57.075397 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a262ac98859b4208b70573662cb898cb" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
I20240502 14:14:57.075619 29443 raft_consensus.cc:3037] T a262ac98859b4208b70573662cb898cb P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:57.075991 29176 raft_consensus.cc:2454] T a262ac98859b4208b70573662cb898cb P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:14:57.076668 29510 leader_election.cc:302] T a262ac98859b4208b70573662cb898cb P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 4092902b3e674857a0147f1b4f2cdb86; no voters:
I20240502 14:14:57.077204 30763 raft_consensus.cc:2781] T a262ac98859b4208b70573662cb898cb P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:57.077373 30763 raft_consensus.cc:686] T a262ac98859b4208b70573662cb898cb P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Becoming Leader. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Running, Role: LEADER
I20240502 14:14:57.077785 30763 consensus_queue.cc:237] T a262ac98859b4208b70573662cb898cb P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:57.078190 30777 ts_tablet_manager.cc:1398] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 14:14:57.078963 29443 raft_consensus.cc:2454] T a262ac98859b4208b70573662cb898cb P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:14:57.079470 30619 leader_election.cc:288] T a262ac98859b4208b70573662cb898cb P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Requested vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:14:57.081445 29443 consensus_queue.cc:237] T 177a34c96aa84cb4aa50bbd79cab4144 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:57.083424 30776 ts_tablet_manager.cc:1398] T 2434830a866e4f7dbe7c8fa18d48e735 P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.033s user 0.000s sys 0.002s
I20240502 14:14:57.084276 30778 tablet_bootstrap.cc:492] T 822e90c09531419e9b32af0438fceb57 P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:14:57.084597 30778 ts_tablet_manager.cc:1367] T 822e90c09531419e9b32af0438fceb57 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.066s user 0.002s sys 0.006s
I20240502 14:14:57.085611 30778 raft_consensus.cc:348] T 822e90c09531419e9b32af0438fceb57 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:57.085942 30778 raft_consensus.cc:374] T 822e90c09531419e9b32af0438fceb57 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:57.086086 30778 raft_consensus.cc:729] T 822e90c09531419e9b32af0438fceb57 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:14:57.086442 30778 consensus_queue.cc:260] T 822e90c09531419e9b32af0438fceb57 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:57.087778 30783 ts_tablet_manager.cc:1888] T 9eb6af02e761412dae5ab888a4b97455 P c1100e16b3b9495ca61d4f95e89b6a89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:57.087941 30783 log.cc:1196] T 9eb6af02e761412dae5ab888a4b97455 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal/wals/9eb6af02e761412dae5ab888a4b97455
I20240502 14:14:57.088495 30783 ts_tablet_manager.cc:1909] T 9eb6af02e761412dae5ab888a4b97455 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting consensus metadata
I20240502 14:14:57.089336 30782 ts_tablet_manager.cc:1888] T 9eb6af02e761412dae5ab888a4b97455 P 89af7f423cf44717a08d6b142f0ed49d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:57.096793 26271 catalog_manager.cc:4830] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): tablet 9eb6af02e761412dae5ab888a4b97455 (table rebalancer_dynamic_table_new_name [id=ec92b676b3e04dcc8943814a451c0c4a]) successfully deleted
I20240502 14:14:57.097332 30782 log.cc:1196] T 9eb6af02e761412dae5ab888a4b97455 P 89af7f423cf44717a08d6b142f0ed49d: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/9eb6af02e761412dae5ab888a4b97455
I20240502 14:14:57.097859 30782 ts_tablet_manager.cc:1909] T 9eb6af02e761412dae5ab888a4b97455 P 89af7f423cf44717a08d6b142f0ed49d: Deleting consensus metadata
W20240502 14:14:57.098955 29378 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): Couldn't send request to peer ba187e38a2754d60a12de9ac4856430a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 177a34c96aa84cb4aa50bbd79cab4144. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:57.099203 26271 catalog_manager.cc:4830] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): tablet 9eb6af02e761412dae5ab888a4b97455 (table rebalancer_dynamic_table_new_name [id=ec92b676b3e04dcc8943814a451c0c4a]) successfully deleted
I20240502 14:14:57.100689 26423 raft_consensus.cc:1261] T 177a34c96aa84cb4aa50bbd79cab4144 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:57.100739 26280 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:14:57.101640 30452 consensus_queue.cc:1035] T 177a34c96aa84cb4aa50bbd79cab4144 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:14:57.110069 29176 raft_consensus.cc:1261] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:57.110886 30452 consensus_queue.cc:1035] T 177a34c96aa84cb4aa50bbd79cab4144 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:14:57.112872 26668 tablet_service.cc:1467] Processing CreateTablet for tablet 14f92995b2e0417687abdc8b35685ef5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b7b2caa9d8a84820b3e17aa9f5484bd9]), partition=
I20240502 14:14:57.113106 26538 tablet_service.cc:1467] Processing CreateTablet for tablet 14f92995b2e0417687abdc8b35685ef5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b7b2caa9d8a84820b3e17aa9f5484bd9]), partition=
I20240502 14:14:57.113313 26668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14f92995b2e0417687abdc8b35685ef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:57.113480 26538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14f92995b2e0417687abdc8b35685ef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:57.117161 29553 tablet_service.cc:1467] Processing CreateTablet for tablet 14f92995b2e0417687abdc8b35685ef5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b7b2caa9d8a84820b3e17aa9f5484bd9]), partition=
I20240502 14:14:57.117595 29553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14f92995b2e0417687abdc8b35685ef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:57.131083 30756 tablet_bootstrap.cc:492] T d1aa4f4be5894ec5a64275f1172e1f55 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:57.131875 30756 tablet_bootstrap.cc:492] T d1aa4f4be5894ec5a64275f1172e1f55 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap complete.
I20240502 14:14:57.132329 30756 ts_tablet_manager.cc:1367] T d1aa4f4be5894ec5a64275f1172e1f55 P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.212s user 0.047s sys 0.008s
I20240502 14:14:57.133534 30756 raft_consensus.cc:348] T d1aa4f4be5894ec5a64275f1172e1f55 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:57.135201 30756 raft_consensus.cc:729] T d1aa4f4be5894ec5a64275f1172e1f55 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:14:57.135738 30776 tablet_bootstrap.cc:492] T 14f92995b2e0417687abdc8b35685ef5 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:14:57.142807 30776 tablet_bootstrap.cc:654] T 14f92995b2e0417687abdc8b35685ef5 P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:57.153038 30756 consensus_queue.cc:260] T d1aa4f4be5894ec5a64275f1172e1f55 P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:57.153681 30452 raft_consensus.cc:2932] T 177a34c96aa84cb4aa50bbd79cab4144 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:14:57.154220 30756 ts_tablet_manager.cc:1398] T d1aa4f4be5894ec5a64275f1172e1f55 P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.022s user 0.000s sys 0.002s
I20240502 14:14:57.155414 26578 tablet_copy_service.cc:342] P ba187e38a2754d60a12de9ac4856430a: Request end of tablet copy session 1cb67158133e4648a6eceba32d4dbba3-d1aa4f4be5894ec5a64275f1172e1f55 received from {username='slave'} at 127.19.94.197:58769
I20240502 14:14:57.155573 26578 tablet_copy_service.cc:433] P ba187e38a2754d60a12de9ac4856430a: ending tablet copy session 1cb67158133e4648a6eceba32d4dbba3-d1aa4f4be5894ec5a64275f1172e1f55 on tablet d1aa4f4be5894ec5a64275f1172e1f55 with peer 1cb67158133e4648a6eceba32d4dbba3
I20240502 14:14:57.157982 26427 raft_consensus.cc:2932] T 177a34c96aa84cb4aa50bbd79cab4144 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:14:57.168267 30803 ts_tablet_manager.cc:902] T c586354e10cb4bedbbb20bd103ad477e P 4092902b3e674857a0147f1b4f2cdb86: Initiating tablet copy from peer ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:14:57.168747 26280 catalog_manager.cc:5484] T 177a34c96aa84cb4aa50bbd79cab4144 P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New cstate: current_term: 1 leader_uuid: "68a3c2b17ed94e06aba66ed4178132ab" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:14:57.169703 30807 tablet_bootstrap.cc:492] T 14f92995b2e0417687abdc8b35685ef5 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:14:57.175796 26280 catalog_manager.cc:5484] T a262ac98859b4208b70573662cb898cb P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200). New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:14:57.176930 30807 tablet_bootstrap.cc:654] T 14f92995b2e0417687abdc8b35685ef5 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:57.191360 29176 raft_consensus.cc:2932] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:14:57.205559 30776 tablet_bootstrap.cc:492] T 14f92995b2e0417687abdc8b35685ef5 P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:14:57.205829 30776 ts_tablet_manager.cc:1367] T 14f92995b2e0417687abdc8b35685ef5 P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.070s user 0.011s sys 0.034s
I20240502 14:14:57.206780 30776 raft_consensus.cc:348] T 14f92995b2e0417687abdc8b35685ef5 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:57.207115 30776 raft_consensus.cc:374] T 14f92995b2e0417687abdc8b35685ef5 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:57.207257 30776 raft_consensus.cc:729] T 14f92995b2e0417687abdc8b35685ef5 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:14:57.207626 30776 consensus_queue.cc:260] T 14f92995b2e0417687abdc8b35685ef5 P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:57.208885 30776 ts_tablet_manager.cc:1398] T 14f92995b2e0417687abdc8b35685ef5 P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:14:57.211902 30803 tablet_copy_client.cc:308] T c586354e10cb4bedbbb20bd103ad477e P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.194:36025
I20240502 14:14:57.217207 30778 ts_tablet_manager.cc:1398] T 822e90c09531419e9b32af0438fceb57 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.132s user 0.030s sys 0.033s
I20240502 14:14:57.186326 30806 tablet_bootstrap.cc:492] T 14f92995b2e0417687abdc8b35685ef5 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:57.212996 26578 tablet_copy_service.cc:140] P ba187e38a2754d60a12de9ac4856430a: Received BeginTabletCopySession request for tablet c586354e10cb4bedbbb20bd103ad477e from peer 4092902b3e674857a0147f1b4f2cdb86 ({username='slave'} at 127.19.94.200:52883)
I20240502 14:14:57.219324 26578 tablet_copy_service.cc:161] P ba187e38a2754d60a12de9ac4856430a: Beginning new tablet copy session on tablet c586354e10cb4bedbbb20bd103ad477e from peer 4092902b3e674857a0147f1b4f2cdb86 at {username='slave'} at 127.19.94.200:52883: session id = 4092902b3e674857a0147f1b4f2cdb86-c586354e10cb4bedbbb20bd103ad477e
I20240502 14:14:57.220608 30807 tablet_bootstrap.cc:492] T 14f92995b2e0417687abdc8b35685ef5 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:14:57.221042 30807 ts_tablet_manager.cc:1367] T 14f92995b2e0417687abdc8b35685ef5 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.051s user 0.005s sys 0.000s
I20240502 14:14:57.222051 30807 raft_consensus.cc:348] T 14f92995b2e0417687abdc8b35685ef5 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:57.222395 30807 raft_consensus.cc:374] T 14f92995b2e0417687abdc8b35685ef5 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:57.222527 30807 raft_consensus.cc:729] T 14f92995b2e0417687abdc8b35685ef5 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:14:57.222862 26578 tablet_copy_source_session.cc:205] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:57.222838 30807 consensus_queue.cc:260] T 14f92995b2e0417687abdc8b35685ef5 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:57.223490 30807 ts_tablet_manager.cc:1398] T 14f92995b2e0417687abdc8b35685ef5 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:57.224637 30806 tablet_bootstrap.cc:654] T 14f92995b2e0417687abdc8b35685ef5 P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:57.232141 30803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c586354e10cb4bedbbb20bd103ad477e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:57.243471 30803 tablet_copy_client.cc:783] T c586354e10cb4bedbbb20bd103ad477e P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:57.243983 30803 tablet_copy_client.cc:647] T c586354e10cb4bedbbb20bd103ad477e P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:57.247455 30713 ts_tablet_manager.cc:902] T 66554eb8b44145d4b474707b061c9659 P c1100e16b3b9495ca61d4f95e89b6a89: Initiating tablet copy from peer ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:14:57.258656 30806 tablet_bootstrap.cc:492] T 14f92995b2e0417687abdc8b35685ef5 P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:57.258988 30806 ts_tablet_manager.cc:1367] T 14f92995b2e0417687abdc8b35685ef5 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.073s user 0.017s sys 0.000s
I20240502 14:14:57.259963 30806 raft_consensus.cc:348] T 14f92995b2e0417687abdc8b35685ef5 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:57.260270 30806 raft_consensus.cc:374] T 14f92995b2e0417687abdc8b35685ef5 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:57.260427 30806 raft_consensus.cc:729] T 14f92995b2e0417687abdc8b35685ef5 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:57.260725 30806 consensus_queue.cc:260] T 14f92995b2e0417687abdc8b35685ef5 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:57.261390 30806 ts_tablet_manager.cc:1398] T 14f92995b2e0417687abdc8b35685ef5 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:14:57.262034 30803 tablet_copy_client.cc:523] T c586354e10cb4bedbbb20bd103ad477e P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:57.266268 30791 raft_consensus.cc:482] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:57.266474 30791 raft_consensus.cc:504] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:57.267315 30791 leader_election.cc:288] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:14:57.268693 29309 raft_consensus.cc:1203] T d7f02f897ce54fefb03832da3c2d95e0 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:57.272064 30615 raft_consensus.cc:482] T 14f92995b2e0417687abdc8b35685ef5 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:57.274106 30803 tablet_bootstrap.cc:492] T c586354e10cb4bedbbb20bd103ad477e P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:14:57.272264 30615 raft_consensus.cc:504] T 14f92995b2e0417687abdc8b35685ef5 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:57.274853 30615 leader_election.cc:288] T 14f92995b2e0417687abdc8b35685ef5 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:14:57.275187 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "822e90c09531419e9b32af0438fceb57" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:14:57.275418 29309 raft_consensus.cc:2454] T 822e90c09531419e9b32af0438fceb57 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:14:57.275851 26362 leader_election.cc:302] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:14:57.275820 26687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14f92995b2e0417687abdc8b35685ef5" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:14:57.276087 26687 raft_consensus.cc:2454] T 14f92995b2e0417687abdc8b35685ef5 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 0.
I20240502 14:14:57.276176 30791 raft_consensus.cc:2781] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:57.276293 30791 raft_consensus.cc:482] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:57.276424 30791 raft_consensus.cc:3037] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:57.277381 26494 leader_election.cc:302] T 14f92995b2e0417687abdc8b35685ef5 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:57.277796 30615 raft_consensus.cc:2781] T 14f92995b2e0417687abdc8b35685ef5 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:57.277902 30615 raft_consensus.cc:482] T 14f92995b2e0417687abdc8b35685ef5 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:57.277997 30615 raft_consensus.cc:3037] T 14f92995b2e0417687abdc8b35685ef5 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:57.279649 30791 raft_consensus.cc:504] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:57.276924 29573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14f92995b2e0417687abdc8b35685ef5" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86" is_pre_election: true
I20240502 14:14:57.280447 29573 raft_consensus.cc:2454] T 14f92995b2e0417687abdc8b35685ef5 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 0.
I20240502 14:14:57.280520 30791 leader_election.cc:288] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Requested vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:14:57.281067 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "822e90c09531419e9b32af0438fceb57" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
I20240502 14:14:57.281308 29309 raft_consensus.cc:3037] T 822e90c09531419e9b32af0438fceb57 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:57.282213 30823 ts_tablet_manager.cc:902] T 533b42de2de14abc96c00a928d5deed1 P 68a3c2b17ed94e06aba66ed4178132ab: Initiating tablet copy from peer e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:57.282810 30615 raft_consensus.cc:504] T 14f92995b2e0417687abdc8b35685ef5 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:57.284067 29573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14f92995b2e0417687abdc8b35685ef5" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86"
I20240502 14:14:57.284138 26687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14f92995b2e0417687abdc8b35685ef5" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:14:57.284281 29573 raft_consensus.cc:3037] T 14f92995b2e0417687abdc8b35685ef5 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:57.284447 26687 raft_consensus.cc:3037] T 14f92995b2e0417687abdc8b35685ef5 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:57.284621 29309 raft_consensus.cc:2454] T 822e90c09531419e9b32af0438fceb57 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:14:57.285203 26362 leader_election.cc:302] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:14:57.285571 30791 raft_consensus.cc:2781] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:57.285753 30791 raft_consensus.cc:686] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Becoming Leader. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Running, Role: LEADER
I20240502 14:14:57.286060 30791 consensus_queue.cc:237] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:14:57.287201 29573 raft_consensus.cc:2454] T 14f92995b2e0417687abdc8b35685ef5 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:14:57.287226 26687 raft_consensus.cc:2454] T 14f92995b2e0417687abdc8b35685ef5 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:14:57.293216 30274 raft_consensus.cc:482] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:57.293404 30274 raft_consensus.cc:504] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:57.293488 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "822e90c09531419e9b32af0438fceb57" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:14:57.293752 29443 raft_consensus.cc:2454] T 822e90c09531419e9b32af0438fceb57 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:14:57.294061 30274 leader_election.cc:288] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:14:57.294736 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "822e90c09531419e9b32af0438fceb57" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
I20240502 14:14:57.294903 29443 raft_consensus.cc:3037] T 822e90c09531419e9b32af0438fceb57 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:57.295560 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2434830a866e4f7dbe7c8fa18d48e735" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:14:57.295787 29309 raft_consensus.cc:2454] T 2434830a866e4f7dbe7c8fa18d48e735 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 0.
I20240502 14:14:57.296197 29111 leader_election.cc:302] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:14:57.296507 30274 raft_consensus.cc:2781] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:57.296604 30274 raft_consensus.cc:482] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:57.296689 30274 raft_consensus.cc:3037] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:57.297791 29443 raft_consensus.cc:2454] T 822e90c09531419e9b32af0438fceb57 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:14:57.283560 30615 leader_election.cc:288] T 14f92995b2e0417687abdc8b35685ef5 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Requested vote from peers 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:14:57.287875 26494 leader_election.cc:302] T 14f92995b2e0417687abdc8b35685ef5 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:57.299629 30615 raft_consensus.cc:2781] T 14f92995b2e0417687abdc8b35685ef5 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:57.300365 30615 raft_consensus.cc:686] T 14f92995b2e0417687abdc8b35685ef5 P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Becoming Leader. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Running, Role: LEADER
I20240502 14:14:57.300863 30615 consensus_queue.cc:237] T 14f92995b2e0417687abdc8b35685ef5 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:14:57.300979 30274 raft_consensus.cc:504] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:57.301743 29575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2434830a866e4f7dbe7c8fa18d48e735" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86" is_pre_election: true
I20240502 14:14:57.302964 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2434830a866e4f7dbe7c8fa18d48e735" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
I20240502 14:14:57.303254 29309 raft_consensus.cc:3037] T 2434830a866e4f7dbe7c8fa18d48e735 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:57.306237 29309 raft_consensus.cc:2454] T 2434830a866e4f7dbe7c8fa18d48e735 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 1.
I20240502 14:14:57.306545 29575 raft_consensus.cc:2454] T 2434830a866e4f7dbe7c8fa18d48e735 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 0.
I20240502 14:14:57.307512 29574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2434830a866e4f7dbe7c8fa18d48e735" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86"
I20240502 14:14:57.307746 29574 raft_consensus.cc:3037] T 2434830a866e4f7dbe7c8fa18d48e735 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:57.307932 30274 leader_election.cc:288] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 election: Requested vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:14:57.309363 29111 leader_election.cc:302] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:14:57.309756 30274 raft_consensus.cc:2781] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:57.311161 29574 raft_consensus.cc:2454] T 2434830a866e4f7dbe7c8fa18d48e735 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 1.
I20240502 14:14:57.309893 30274 raft_consensus.cc:686] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 LEADER]: Becoming Leader. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Running, Role: LEADER
I20240502 14:14:57.313258 30274 consensus_queue.cc:237] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:57.315697 30713 tablet_copy_client.cc:308] T 66554eb8b44145d4b474707b061c9659 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.194:36025
I20240502 14:14:57.317365 26578 tablet_copy_service.cc:140] P ba187e38a2754d60a12de9ac4856430a: Received BeginTabletCopySession request for tablet 66554eb8b44145d4b474707b061c9659 from peer c1100e16b3b9495ca61d4f95e89b6a89 ({username='slave'} at 127.19.94.198:36021)
I20240502 14:14:57.317632 26578 tablet_copy_service.cc:161] P ba187e38a2754d60a12de9ac4856430a: Beginning new tablet copy session on tablet 66554eb8b44145d4b474707b061c9659 from peer c1100e16b3b9495ca61d4f95e89b6a89 at {username='slave'} at 127.19.94.198:36021: session id = c1100e16b3b9495ca61d4f95e89b6a89-66554eb8b44145d4b474707b061c9659
I20240502 14:14:57.321650 26276 catalog_manager.cc:5484] T 14f92995b2e0417687abdc8b35685ef5 P ba187e38a2754d60a12de9ac4856430a reported cstate change: term changed from 0 to 1, leader changed from <none> to ba187e38a2754d60a12de9ac4856430a (127.19.94.194). New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:14:57.323750 29176 raft_consensus.cc:1203] T d1aa4f4be5894ec5a64275f1172e1f55 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:57.327478 26578 tablet_copy_source_session.cc:205] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:57.329753 30713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66554eb8b44145d4b474707b061c9659. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:57.330121 26276 catalog_manager.cc:5484] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: term changed from 0 to 1, leader changed from <none> to 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193). New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:14:57.363584 26280 catalog_manager.cc:5484] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197). New cstate: current_term: 1 leader_uuid: "1cb67158133e4648a6eceba32d4dbba3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:14:57.367475 30756 ts_tablet_manager.cc:902] T f8923ed1ce94441a8eb303b6f7dd3acb P 1cb67158133e4648a6eceba32d4dbba3: Initiating tablet copy from peer 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:14:57.369760 30827 ts_tablet_manager.cc:902] T 62218b2a597e4fb188ced6d2b9a83a59 P 4092902b3e674857a0147f1b4f2cdb86: Initiating tablet copy from peer ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:14:57.388805 30827 tablet_copy_client.cc:308] T 62218b2a597e4fb188ced6d2b9a83a59 P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.194:36025
I20240502 14:14:57.390138 26578 tablet_copy_service.cc:140] P ba187e38a2754d60a12de9ac4856430a: Received BeginTabletCopySession request for tablet 62218b2a597e4fb188ced6d2b9a83a59 from peer 4092902b3e674857a0147f1b4f2cdb86 ({username='slave'} at 127.19.94.200:52883)
I20240502 14:14:57.390367 26578 tablet_copy_service.cc:161] P ba187e38a2754d60a12de9ac4856430a: Beginning new tablet copy session on tablet 62218b2a597e4fb188ced6d2b9a83a59 from peer 4092902b3e674857a0147f1b4f2cdb86 at {username='slave'} at 127.19.94.200:52883: session id = 4092902b3e674857a0147f1b4f2cdb86-62218b2a597e4fb188ced6d2b9a83a59
I20240502 14:14:57.394037 26578 tablet_copy_source_session.cc:205] T 62218b2a597e4fb188ced6d2b9a83a59 P ba187e38a2754d60a12de9ac4856430a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:57.395798 30827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62218b2a597e4fb188ced6d2b9a83a59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:57.400516 30713 tablet_copy_client.cc:783] T 66554eb8b44145d4b474707b061c9659 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:57.402554 30713 tablet_copy_client.cc:647] T 66554eb8b44145d4b474707b061c9659 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:57.406701 30756 tablet_copy_client.cc:308] T f8923ed1ce94441a8eb303b6f7dd3acb P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.200:46097
I20240502 14:14:57.407989 29594 tablet_copy_service.cc:140] P 4092902b3e674857a0147f1b4f2cdb86: Received BeginTabletCopySession request for tablet f8923ed1ce94441a8eb303b6f7dd3acb from peer 1cb67158133e4648a6eceba32d4dbba3 ({username='slave'} at 127.19.94.197:54421)
I20240502 14:14:57.408218 29594 tablet_copy_service.cc:161] P 4092902b3e674857a0147f1b4f2cdb86: Beginning new tablet copy session on tablet f8923ed1ce94441a8eb303b6f7dd3acb from peer 1cb67158133e4648a6eceba32d4dbba3 at {username='slave'} at 127.19.94.197:54421: session id = 1cb67158133e4648a6eceba32d4dbba3-f8923ed1ce94441a8eb303b6f7dd3acb
I20240502 14:14:57.411213 29594 tablet_copy_source_session.cc:205] T f8923ed1ce94441a8eb303b6f7dd3acb P 4092902b3e674857a0147f1b4f2cdb86: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:57.412528 30713 tablet_copy_client.cc:523] T 66554eb8b44145d4b474707b061c9659 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:57.412809 30756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8923ed1ce94441a8eb303b6f7dd3acb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:57.430682 30756 tablet_copy_client.cc:783] T f8923ed1ce94441a8eb303b6f7dd3acb P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:57.431197 30756 tablet_copy_client.cc:647] T f8923ed1ce94441a8eb303b6f7dd3acb P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:57.433699 30756 tablet_copy_client.cc:523] T f8923ed1ce94441a8eb303b6f7dd3acb P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:57.440356 30756 tablet_bootstrap.cc:492] T f8923ed1ce94441a8eb303b6f7dd3acb P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:14:57.444362 30713 tablet_bootstrap.cc:492] T 66554eb8b44145d4b474707b061c9659 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:14:57.445462 26280 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:14:57.449476 30827 tablet_copy_client.cc:783] T 62218b2a597e4fb188ced6d2b9a83a59 P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:57.450013 30827 tablet_copy_client.cc:647] T 62218b2a597e4fb188ced6d2b9a83a59 P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:57.457232 30827 tablet_copy_client.cc:523] T 62218b2a597e4fb188ced6d2b9a83a59 P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:57.465584 30827 tablet_bootstrap.cc:492] T 62218b2a597e4fb188ced6d2b9a83a59 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:14:57.466945 30823 tablet_copy_client.cc:308] T 533b42de2de14abc96c00a928d5deed1 P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.196:32843
I20240502 14:14:57.467761 26839 tablet_copy_service.cc:140] P e4cfb8fc5a474177af6848a6efa13a27: Received BeginTabletCopySession request for tablet 533b42de2de14abc96c00a928d5deed1 from peer 68a3c2b17ed94e06aba66ed4178132ab ({username='slave'} at 127.19.94.199:38699)
I20240502 14:14:57.467968 26839 tablet_copy_service.cc:161] P e4cfb8fc5a474177af6848a6efa13a27: Beginning new tablet copy session on tablet 533b42de2de14abc96c00a928d5deed1 from peer 68a3c2b17ed94e06aba66ed4178132ab at {username='slave'} at 127.19.94.199:38699: session id = 68a3c2b17ed94e06aba66ed4178132ab-533b42de2de14abc96c00a928d5deed1
I20240502 14:14:57.471232 26839 tablet_copy_source_session.cc:205] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:57.472481 30763 consensus_queue.cc:1035] T a262ac98859b4208b70573662cb898cb P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 14:14:57.475757 30823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 533b42de2de14abc96c00a928d5deed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:57.492877 30823 tablet_copy_client.cc:783] T 533b42de2de14abc96c00a928d5deed1 P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:57.492847 30756 tablet_bootstrap.cc:492] T f8923ed1ce94441a8eb303b6f7dd3acb P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:57.495019 30756 tablet_bootstrap.cc:492] T f8923ed1ce94441a8eb303b6f7dd3acb P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap complete.
I20240502 14:14:57.495869 30756 ts_tablet_manager.cc:1367] T f8923ed1ce94441a8eb303b6f7dd3acb P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.056s user 0.005s sys 0.008s
I20240502 14:14:57.497292 30756 raft_consensus.cc:348] T f8923ed1ce94441a8eb303b6f7dd3acb P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:57.499379 26687 raft_consensus.cc:1261] T 14f92995b2e0417687abdc8b35685ef5 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:57.500099 30615 consensus_queue.cc:1035] T 14f92995b2e0417687abdc8b35685ef5 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:57.506229 29573 raft_consensus.cc:1261] T 14f92995b2e0417687abdc8b35685ef5 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:57.507901 30615 consensus_queue.cc:1035] T 14f92995b2e0417687abdc8b35685ef5 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:57.509052 30803 tablet_bootstrap.cc:492] T c586354e10cb4bedbbb20bd103ad477e P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:57.509806 30803 tablet_bootstrap.cc:492] T c586354e10cb4bedbbb20bd103ad477e P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap complete.
I20240502 14:14:57.510272 30803 ts_tablet_manager.cc:1367] T c586354e10cb4bedbbb20bd103ad477e P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.236s user 0.052s sys 0.005s
I20240502 14:14:57.510780 30823 tablet_copy_client.cc:647] T 533b42de2de14abc96c00a928d5deed1 P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:57.512135 30803 raft_consensus.cc:348] T c586354e10cb4bedbbb20bd103ad477e P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:57.523949 30756 raft_consensus.cc:729] T f8923ed1ce94441a8eb303b6f7dd3acb P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:14:57.524884 30756 consensus_queue.cc:260] T f8923ed1ce94441a8eb303b6f7dd3acb P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:57.527272 30756 ts_tablet_manager.cc:1398] T f8923ed1ce94441a8eb303b6f7dd3acb P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.031s user 0.000s sys 0.002s
I20240502 14:14:57.553516 26280 catalog_manager.cc:5576] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325) does not have the latest schema for tablet 14f92995b2e0417687abdc8b35685ef5 (table rebalancer_dynamic_table [id=b7b2caa9d8a84820b3e17aa9f5484bd9]). Expected version 1 got 0
I20240502 14:14:57.559439 30823 tablet_copy_client.cc:523] T 533b42de2de14abc96c00a928d5deed1 P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:57.531549 29594 tablet_copy_service.cc:342] P 4092902b3e674857a0147f1b4f2cdb86: Request end of tablet copy session 1cb67158133e4648a6eceba32d4dbba3-f8923ed1ce94441a8eb303b6f7dd3acb received from {username='slave'} at 127.19.94.197:54421
I20240502 14:14:57.563943 29594 tablet_copy_service.cc:433] P 4092902b3e674857a0147f1b4f2cdb86: ending tablet copy session 1cb67158133e4648a6eceba32d4dbba3-f8923ed1ce94441a8eb303b6f7dd3acb on tablet f8923ed1ce94441a8eb303b6f7dd3acb with peer 1cb67158133e4648a6eceba32d4dbba3
I20240502 14:14:57.565274 30803 raft_consensus.cc:729] T c586354e10cb4bedbbb20bd103ad477e P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:14:57.569914 30803 consensus_queue.cc:260] T c586354e10cb4bedbbb20bd103ad477e P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:57.571270 30803 ts_tablet_manager.cc:1398] T c586354e10cb4bedbbb20bd103ad477e P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.060s user 0.002s sys 0.000s
I20240502 14:14:57.576263 30619 consensus_queue.cc:1035] T a262ac98859b4208b70573662cb898cb P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:57.585502 26280 catalog_manager.cc:5576] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097) does not have the latest schema for tablet 14f92995b2e0417687abdc8b35685ef5 (table rebalancer_dynamic_table [id=b7b2caa9d8a84820b3e17aa9f5484bd9]). Expected version 1 got 0
I20240502 14:14:57.590453 26578 tablet_copy_service.cc:342] P ba187e38a2754d60a12de9ac4856430a: Request end of tablet copy session 4092902b3e674857a0147f1b4f2cdb86-c586354e10cb4bedbbb20bd103ad477e received from {username='slave'} at 127.19.94.200:52883
I20240502 14:14:57.590637 26578 tablet_copy_service.cc:433] P ba187e38a2754d60a12de9ac4856430a: ending tablet copy session 4092902b3e674857a0147f1b4f2cdb86-c586354e10cb4bedbbb20bd103ad477e on tablet c586354e10cb4bedbbb20bd103ad477e with peer 4092902b3e674857a0147f1b4f2cdb86
I20240502 14:14:57.595968 30823 tablet_bootstrap.cc:492] T 533b42de2de14abc96c00a928d5deed1 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:14:57.612259 30870 tablet.cc:1722] T 14f92995b2e0417687abdc8b35685ef5 P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:57.613076 30870 tablet.cc:1620] T 14f92995b2e0417687abdc8b35685ef5 P 4092902b3e674857a0147f1b4f2cdb86: MemRowSet was empty: no flush needed.
I20240502 14:14:57.615830 26276 catalog_manager.cc:5576] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097) does not have the latest schema for tablet 14f92995b2e0417687abdc8b35685ef5 (table rebalancer_dynamic_table [id=b7b2caa9d8a84820b3e17aa9f5484bd9]). Expected version 1 got 0
I20240502 14:14:57.628034 26276 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=b7b2caa9d8a84820b3e17aa9f5484bd9] alter complete (version 1)
I20240502 14:14:57.639796 29573 raft_consensus.cc:1203] T c586354e10cb4bedbbb20bd103ad477e P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:57.641306 30864 tablet.cc:1722] T 14f92995b2e0417687abdc8b35685ef5 P ba187e38a2754d60a12de9ac4856430a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:57.642467 30864 tablet.cc:1620] T 14f92995b2e0417687abdc8b35685ef5 P ba187e38a2754d60a12de9ac4856430a: MemRowSet was empty: no flush needed.
I20240502 14:14:57.646576 30865 tablet.cc:1722] T 14f92995b2e0417687abdc8b35685ef5 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:57.647755 30865 tablet.cc:1620] T 14f92995b2e0417687abdc8b35685ef5 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: MemRowSet was empty: no flush needed.
I20240502 14:14:57.650910 26280 catalog_manager.cc:5576] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325) does not have the latest schema for tablet 14f92995b2e0417687abdc8b35685ef5 (table rebalancer_dynamic_table [id=b7b2caa9d8a84820b3e17aa9f5484bd9]). Expected version 1 got 0
I20240502 14:14:57.652148 30871 ts_tablet_manager.cc:902] T 177a34c96aa84cb4aa50bbd79cab4144 P ba187e38a2754d60a12de9ac4856430a: Initiating tablet copy from peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:14:57.684577 30871 tablet_copy_client.cc:308] T 177a34c96aa84cb4aa50bbd79cab4144 P ba187e38a2754d60a12de9ac4856430a: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.199:41781
I20240502 14:14:57.686312 29463 tablet_copy_service.cc:140] P 68a3c2b17ed94e06aba66ed4178132ab: Received BeginTabletCopySession request for tablet 177a34c96aa84cb4aa50bbd79cab4144 from peer ba187e38a2754d60a12de9ac4856430a ({username='slave'} at 127.19.94.194:57813)
I20240502 14:14:57.686699 29463 tablet_copy_service.cc:161] P 68a3c2b17ed94e06aba66ed4178132ab: Beginning new tablet copy session on tablet 177a34c96aa84cb4aa50bbd79cab4144 from peer ba187e38a2754d60a12de9ac4856430a at {username='slave'} at 127.19.94.194:57813: session id = ba187e38a2754d60a12de9ac4856430a-177a34c96aa84cb4aa50bbd79cab4144
I20240502 14:14:57.688860 30823 tablet_bootstrap.cc:492] T 533b42de2de14abc96c00a928d5deed1 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:57.689437 30823 tablet_bootstrap.cc:492] T 533b42de2de14abc96c00a928d5deed1 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap complete.
I20240502 14:14:57.689788 30823 ts_tablet_manager.cc:1367] T 533b42de2de14abc96c00a928d5deed1 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.094s user 0.054s sys 0.007s
I20240502 14:14:57.690699 30823 raft_consensus.cc:348] T 533b42de2de14abc96c00a928d5deed1 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:57.690973 30823 raft_consensus.cc:729] T 533b42de2de14abc96c00a928d5deed1 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:14:57.691277 30823 consensus_queue.cc:260] T 533b42de2de14abc96c00a928d5deed1 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:57.693081 30823 ts_tablet_manager.cc:1398] T 533b42de2de14abc96c00a928d5deed1 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240502 14:14:57.694901 26839 tablet_copy_service.cc:342] P e4cfb8fc5a474177af6848a6efa13a27: Request end of tablet copy session 68a3c2b17ed94e06aba66ed4178132ab-533b42de2de14abc96c00a928d5deed1 received from {username='slave'} at 127.19.94.199:38699
I20240502 14:14:57.695065 26839 tablet_copy_service.cc:433] P e4cfb8fc5a474177af6848a6efa13a27: ending tablet copy session 68a3c2b17ed94e06aba66ed4178132ab-533b42de2de14abc96c00a928d5deed1 on tablet 533b42de2de14abc96c00a928d5deed1 with peer 68a3c2b17ed94e06aba66ed4178132ab
I20240502 14:14:57.692591 29463 tablet_copy_source_session.cc:205] T 177a34c96aa84cb4aa50bbd79cab4144 P 68a3c2b17ed94e06aba66ed4178132ab: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:57.698851 30871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 177a34c96aa84cb4aa50bbd79cab4144. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:57.706202 30830 consensus_queue.cc:1035] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:57.707538 30871 tablet_copy_client.cc:783] T 177a34c96aa84cb4aa50bbd79cab4144 P ba187e38a2754d60a12de9ac4856430a: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:57.707980 30871 tablet_copy_client.cc:647] T 177a34c96aa84cb4aa50bbd79cab4144 P ba187e38a2754d60a12de9ac4856430a: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:57.710575 30871 tablet_copy_client.cc:523] T 177a34c96aa84cb4aa50bbd79cab4144 P ba187e38a2754d60a12de9ac4856430a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:57.715768 30871 tablet_bootstrap.cc:492] T 177a34c96aa84cb4aa50bbd79cab4144 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:57.723591 30165 consensus_queue.cc:1035] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:57.726903 30515 consensus_queue.cc:1035] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:57.730427 30713 tablet_bootstrap.cc:492] T 66554eb8b44145d4b474707b061c9659 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:57.731107 30713 tablet_bootstrap.cc:492] T 66554eb8b44145d4b474707b061c9659 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap complete.
I20240502 14:14:57.731418 30713 ts_tablet_manager.cc:1367] T 66554eb8b44145d4b474707b061c9659 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.288s user 0.048s sys 0.009s
I20240502 14:14:57.732159 30713 raft_consensus.cc:348] T 66554eb8b44145d4b474707b061c9659 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:57.732373 30713 raft_consensus.cc:729] T 66554eb8b44145d4b474707b061c9659 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:14:57.732679 30713 consensus_queue.cc:260] T 66554eb8b44145d4b474707b061c9659 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:57.733443 30713 ts_tablet_manager.cc:1398] T 66554eb8b44145d4b474707b061c9659 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.002s user 0.002s sys 0.002s
I20240502 14:14:57.734375 26578 tablet_copy_service.cc:342] P ba187e38a2754d60a12de9ac4856430a: Request end of tablet copy session c1100e16b3b9495ca61d4f95e89b6a89-66554eb8b44145d4b474707b061c9659 received from {username='slave'} at 127.19.94.198:36021
I20240502 14:14:57.734591 26578 tablet_copy_service.cc:433] P ba187e38a2754d60a12de9ac4856430a: ending tablet copy session c1100e16b3b9495ca61d4f95e89b6a89-66554eb8b44145d4b474707b061c9659 on tablet 66554eb8b44145d4b474707b061c9659 with peer c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:14:57.740232 30827 tablet_bootstrap.cc:492] T 62218b2a597e4fb188ced6d2b9a83a59 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:57.740767 30827 tablet_bootstrap.cc:492] T 62218b2a597e4fb188ced6d2b9a83a59 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap complete.
I20240502 14:14:57.741076 30827 ts_tablet_manager.cc:1367] T 62218b2a597e4fb188ced6d2b9a83a59 P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.276s user 0.050s sys 0.010s
I20240502 14:14:57.742372 30827 raft_consensus.cc:348] T 62218b2a597e4fb188ced6d2b9a83a59 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:57.742667 30827 raft_consensus.cc:729] T 62218b2a597e4fb188ced6d2b9a83a59 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:14:57.743063 30827 consensus_queue.cc:260] T 62218b2a597e4fb188ced6d2b9a83a59 P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:57.743716 30827 ts_tablet_manager.cc:1398] T 62218b2a597e4fb188ced6d2b9a83a59 P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:57.747346 26276 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:14:57.749480 30871 tablet_bootstrap.cc:492] T 177a34c96aa84cb4aa50bbd79cab4144 P ba187e38a2754d60a12de9ac4856430a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:57.749789 29443 raft_consensus.cc:1203] T 533b42de2de14abc96c00a928d5deed1 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:57.750550 30871 tablet_bootstrap.cc:492] T 177a34c96aa84cb4aa50bbd79cab4144 P ba187e38a2754d60a12de9ac4856430a: Bootstrap complete.
I20240502 14:14:57.750823 30871 ts_tablet_manager.cc:1367] T 177a34c96aa84cb4aa50bbd79cab4144 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.035s user 0.012s sys 0.006s
I20240502 14:14:57.751557 26578 tablet_copy_service.cc:342] P ba187e38a2754d60a12de9ac4856430a: Request end of tablet copy session 4092902b3e674857a0147f1b4f2cdb86-62218b2a597e4fb188ced6d2b9a83a59 received from {username='slave'} at 127.19.94.200:52883
I20240502 14:14:57.751689 26578 tablet_copy_service.cc:433] P ba187e38a2754d60a12de9ac4856430a: ending tablet copy session 4092902b3e674857a0147f1b4f2cdb86-62218b2a597e4fb188ced6d2b9a83a59 on tablet 62218b2a597e4fb188ced6d2b9a83a59 with peer 4092902b3e674857a0147f1b4f2cdb86
I20240502 14:14:57.752034 30871 raft_consensus.cc:348] T 177a34c96aa84cb4aa50bbd79cab4144 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:57.752270 30871 raft_consensus.cc:729] T 177a34c96aa84cb4aa50bbd79cab4144 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:57.754662 30165 consensus_queue.cc:1035] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:57.754930 30871 consensus_queue.cc:260] T 177a34c96aa84cb4aa50bbd79cab4144 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:57.764518 30864 tablet.cc:1722] T 14f92995b2e0417687abdc8b35685ef5 P ba187e38a2754d60a12de9ac4856430a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:57.765136 30864 tablet.cc:1620] T 14f92995b2e0417687abdc8b35685ef5 P ba187e38a2754d60a12de9ac4856430a: MemRowSet was empty: no flush needed.
I20240502 14:14:57.765619 30865 tablet.cc:1722] T 14f92995b2e0417687abdc8b35685ef5 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:57.766144 30865 tablet.cc:1620] T 14f92995b2e0417687abdc8b35685ef5 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: MemRowSet was empty: no flush needed.
I20240502 14:14:57.771962 30870 tablet.cc:1722] T 14f92995b2e0417687abdc8b35685ef5 P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:57.772644 30870 tablet.cc:1620] T 14f92995b2e0417687abdc8b35685ef5 P 4092902b3e674857a0147f1b4f2cdb86: MemRowSet was empty: no flush needed.
I20240502 14:14:57.786536 26285 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=b7b2caa9d8a84820b3e17aa9f5484bd9] alter complete (version 2)
I20240502 14:14:57.786638 29573 raft_consensus.cc:1203] T 62218b2a597e4fb188ced6d2b9a83a59 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:57.788537 29174 raft_consensus.cc:1203] T f8923ed1ce94441a8eb303b6f7dd3acb P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:14:57.799943 30871 ts_tablet_manager.cc:1398] T 177a34c96aa84cb4aa50bbd79cab4144 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.049s user 0.000s sys 0.002s
I20240502 14:14:57.806821 29463 tablet_copy_service.cc:342] P 68a3c2b17ed94e06aba66ed4178132ab: Request end of tablet copy session ba187e38a2754d60a12de9ac4856430a-177a34c96aa84cb4aa50bbd79cab4144 received from {username='slave'} at 127.19.94.194:57813
I20240502 14:14:57.807035 29463 tablet_copy_service.cc:433] P 68a3c2b17ed94e06aba66ed4178132ab: ending tablet copy session ba187e38a2754d60a12de9ac4856430a-177a34c96aa84cb4aa50bbd79cab4144 on tablet 177a34c96aa84cb4aa50bbd79cab4144 with peer ba187e38a2754d60a12de9ac4856430a
I20240502 14:14:57.900535 26280 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:14:57.902563 29154 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:14:57.931170 26406 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:14:57.937556 29553 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:14:57.949695 29287 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:57.960484 26668 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:14:57.966392 26280 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_extra_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:57.976840 30864 tablet.cc:1722] T 14f92995b2e0417687abdc8b35685ef5 P ba187e38a2754d60a12de9ac4856430a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:57.979147 29287 tablet_service.cc:1467] Processing CreateTablet for tablet eb065120d1304a2a81ce932934b911f3 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=800c8a30773545b3a7670200a5873c0e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:57.979621 29287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb065120d1304a2a81ce932934b911f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:57.981263 29154 tablet_service.cc:1467] Processing CreateTablet for tablet eb065120d1304a2a81ce932934b911f3 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=800c8a30773545b3a7670200a5873c0e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:57.981696 29154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb065120d1304a2a81ce932934b911f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:57.986694 26797 tablet_service.cc:1467] Processing CreateTablet for tablet 929a85cbe95746b798a7510b3a1cc19e (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=800c8a30773545b3a7670200a5873c0e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:57.987143 26797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 929a85cbe95746b798a7510b3a1cc19e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:57.988711 29422 tablet_service.cc:1467] Processing CreateTablet for tablet eb065120d1304a2a81ce932934b911f3 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=800c8a30773545b3a7670200a5873c0e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:57.989164 29422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb065120d1304a2a81ce932934b911f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:57.993055 29156 tablet_service.cc:1467] Processing CreateTablet for tablet 929a85cbe95746b798a7510b3a1cc19e (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=800c8a30773545b3a7670200a5873c0e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:57.993533 29156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 929a85cbe95746b798a7510b3a1cc19e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:57.995688 29288 tablet_service.cc:1467] Processing CreateTablet for tablet 929a85cbe95746b798a7510b3a1cc19e (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=800c8a30773545b3a7670200a5873c0e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:57.996140 29288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 929a85cbe95746b798a7510b3a1cc19e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:57.998406 29553 tablet_service.cc:1467] Processing CreateTablet for tablet 33a3639c26bf4b35a6e15d6ef173751c (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=800c8a30773545b3a7670200a5873c0e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:57.998895 29553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33a3639c26bf4b35a6e15d6ef173751c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:57.999027 29289 tablet_service.cc:1467] Processing CreateTablet for tablet 33a3639c26bf4b35a6e15d6ef173751c (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=800c8a30773545b3a7670200a5873c0e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:57.999439 29289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33a3639c26bf4b35a6e15d6ef173751c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:58.013159 30923 tablet_bootstrap.cc:492] T 929a85cbe95746b798a7510b3a1cc19e P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:14:58.018959 30925 tablet_bootstrap.cc:492] T eb065120d1304a2a81ce932934b911f3 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:14:58.019292 29423 tablet_service.cc:1467] Processing CreateTablet for tablet 33a3639c26bf4b35a6e15d6ef173751c (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=800c8a30773545b3a7670200a5873c0e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:58.019820 29423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33a3639c26bf4b35a6e15d6ef173751c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:58.023936 30925 tablet_bootstrap.cc:654] T eb065120d1304a2a81ce932934b911f3 P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:58.021696 30924 tablet_bootstrap.cc:492] T 929a85cbe95746b798a7510b3a1cc19e P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:14:58.028342 30923 tablet_bootstrap.cc:654] T 929a85cbe95746b798a7510b3a1cc19e P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:58.028997 30870 tablet.cc:1722] T 14f92995b2e0417687abdc8b35685ef5 P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:58.039865 30865 tablet.cc:1722] T 14f92995b2e0417687abdc8b35685ef5 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:58.050350 30924 tablet_bootstrap.cc:654] T 929a85cbe95746b798a7510b3a1cc19e P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:58.060052 26280 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table_new_name [id=b7b2caa9d8a84820b3e17aa9f5484bd9] alter complete (version 3)
I20240502 14:14:58.070976 30930 tablet_bootstrap.cc:492] T 929a85cbe95746b798a7510b3a1cc19e P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:58.083509 30923 tablet_bootstrap.cc:492] T 929a85cbe95746b798a7510b3a1cc19e P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:14:58.083822 30923 ts_tablet_manager.cc:1367] T 929a85cbe95746b798a7510b3a1cc19e P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.071s user 0.004s sys 0.004s
I20240502 14:14:58.084168 26538 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:14:58.084978 30923 raft_consensus.cc:348] T 929a85cbe95746b798a7510b3a1cc19e P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:58.085314 30923 raft_consensus.cc:374] T 929a85cbe95746b798a7510b3a1cc19e P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:58.089758 30924 tablet_bootstrap.cc:492] T 929a85cbe95746b798a7510b3a1cc19e P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:14:58.090071 30924 ts_tablet_manager.cc:1367] T 929a85cbe95746b798a7510b3a1cc19e P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.068s user 0.006s sys 0.000s
I20240502 14:14:58.091146 30930 tablet_bootstrap.cc:654] T 929a85cbe95746b798a7510b3a1cc19e P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:58.106446 29423 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:14:58.109939 30925 tablet_bootstrap.cc:492] T eb065120d1304a2a81ce932934b911f3 P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:14:58.110217 30925 ts_tablet_manager.cc:1367] T eb065120d1304a2a81ce932934b911f3 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.091s user 0.008s sys 0.000s
I20240502 14:14:58.122135 30925 raft_consensus.cc:348] T eb065120d1304a2a81ce932934b911f3 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:58.122558 30925 raft_consensus.cc:374] T eb065120d1304a2a81ce932934b911f3 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:58.122743 30925 raft_consensus.cc:729] T eb065120d1304a2a81ce932934b911f3 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:14:58.123126 30925 consensus_queue.cc:260] T eb065120d1304a2a81ce932934b911f3 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:58.123912 30925 ts_tablet_manager.cc:1398] T eb065120d1304a2a81ce932934b911f3 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240502 14:14:58.124199 30925 tablet_bootstrap.cc:492] T 33a3639c26bf4b35a6e15d6ef173751c P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:14:58.091713 30924 raft_consensus.cc:348] T 929a85cbe95746b798a7510b3a1cc19e P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:58.126112 30925 tablet_bootstrap.cc:654] T 33a3639c26bf4b35a6e15d6ef173751c P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:58.127450 30924 raft_consensus.cc:374] T 929a85cbe95746b798a7510b3a1cc19e P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:58.127614 30924 raft_consensus.cc:729] T 929a85cbe95746b798a7510b3a1cc19e P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:14:58.127951 30924 consensus_queue.cc:260] T 929a85cbe95746b798a7510b3a1cc19e P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:58.128962 30924 ts_tablet_manager.cc:1398] T 929a85cbe95746b798a7510b3a1cc19e P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.039s user 0.002s sys 0.000s
I20240502 14:14:58.135452 30925 tablet_bootstrap.cc:492] T 33a3639c26bf4b35a6e15d6ef173751c P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:14:58.135731 30925 ts_tablet_manager.cc:1367] T 33a3639c26bf4b35a6e15d6ef173751c P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20240502 14:14:58.137635 30925 raft_consensus.cc:348] T 33a3639c26bf4b35a6e15d6ef173751c P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:58.137982 30925 raft_consensus.cc:374] T 33a3639c26bf4b35a6e15d6ef173751c P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:58.138130 30925 raft_consensus.cc:729] T 33a3639c26bf4b35a6e15d6ef173751c P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:14:58.138612 30925 consensus_queue.cc:260] T 33a3639c26bf4b35a6e15d6ef173751c P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:58.138402 30923 raft_consensus.cc:729] T 929a85cbe95746b798a7510b3a1cc19e P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:14:58.139940 30923 consensus_queue.cc:260] T 929a85cbe95746b798a7510b3a1cc19e P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:58.140705 30925 ts_tablet_manager.cc:1398] T 33a3639c26bf4b35a6e15d6ef173751c P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:14:58.144371 30923 ts_tablet_manager.cc:1398] T 929a85cbe95746b798a7510b3a1cc19e P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.060s user 0.002s sys 0.000s
I20240502 14:14:58.148561 30923 tablet_bootstrap.cc:492] T eb065120d1304a2a81ce932934b911f3 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:14:58.149969 30923 tablet_bootstrap.cc:654] T eb065120d1304a2a81ce932934b911f3 P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:58.152429 30924 tablet_bootstrap.cc:492] T 33a3639c26bf4b35a6e15d6ef173751c P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:14:58.155681 30924 tablet_bootstrap.cc:654] T 33a3639c26bf4b35a6e15d6ef173751c P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:58.166019 30923 tablet_bootstrap.cc:492] T eb065120d1304a2a81ce932934b911f3 P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:14:58.166174 30924 tablet_bootstrap.cc:492] T 33a3639c26bf4b35a6e15d6ef173751c P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:14:58.166368 30923 ts_tablet_manager.cc:1367] T eb065120d1304a2a81ce932934b911f3 P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20240502 14:14:58.166391 30924 ts_tablet_manager.cc:1367] T 33a3639c26bf4b35a6e15d6ef173751c P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.003s
I20240502 14:14:58.167389 30923 raft_consensus.cc:348] T eb065120d1304a2a81ce932934b911f3 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:58.167747 30923 raft_consensus.cc:374] T eb065120d1304a2a81ce932934b911f3 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:58.167886 30923 raft_consensus.cc:729] T eb065120d1304a2a81ce932934b911f3 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:14:58.168417 30930 tablet_bootstrap.cc:492] T 929a85cbe95746b798a7510b3a1cc19e P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:58.168262 30923 consensus_queue.cc:260] T eb065120d1304a2a81ce932934b911f3 P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:58.168591 30930 ts_tablet_manager.cc:1367] T 929a85cbe95746b798a7510b3a1cc19e P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.098s user 0.004s sys 0.001s
I20240502 14:14:58.168746 30931 tablet_bootstrap.cc:492] T 33a3639c26bf4b35a6e15d6ef173751c P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:14:58.169677 30930 raft_consensus.cc:348] T 929a85cbe95746b798a7510b3a1cc19e P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:58.169924 30930 raft_consensus.cc:374] T 929a85cbe95746b798a7510b3a1cc19e P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:58.170042 30930 raft_consensus.cc:729] T 929a85cbe95746b798a7510b3a1cc19e P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:58.170392 30930 consensus_queue.cc:260] T 929a85cbe95746b798a7510b3a1cc19e P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:58.171090 30930 ts_tablet_manager.cc:1398] T 929a85cbe95746b798a7510b3a1cc19e P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:14:58.171702 30924 raft_consensus.cc:348] T 33a3639c26bf4b35a6e15d6ef173751c P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:58.172470 30924 raft_consensus.cc:374] T 33a3639c26bf4b35a6e15d6ef173751c P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:58.172603 30924 raft_consensus.cc:729] T 33a3639c26bf4b35a6e15d6ef173751c P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:14:58.172665 30931 tablet_bootstrap.cc:654] T 33a3639c26bf4b35a6e15d6ef173751c P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:58.172957 30924 consensus_queue.cc:260] T 33a3639c26bf4b35a6e15d6ef173751c P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:58.173619 30924 ts_tablet_manager.cc:1398] T 33a3639c26bf4b35a6e15d6ef173751c P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:14:58.173986 30924 tablet_bootstrap.cc:492] T eb065120d1304a2a81ce932934b911f3 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:14:58.174353 30923 ts_tablet_manager.cc:1398] T eb065120d1304a2a81ce932934b911f3 P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 14:14:58.175498 29309 raft_consensus.cc:1203] T 66554eb8b44145d4b474707b061c9659 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:58.176098 30924 tablet_bootstrap.cc:654] T eb065120d1304a2a81ce932934b911f3 P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
W20240502 14:14:58.182572 26625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.196:32843, user_credentials={real_user=slave}} blocked reactor thread for 157443us
I20240502 14:14:58.184859 26797 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:14:58.195185 30924 tablet_bootstrap.cc:492] T eb065120d1304a2a81ce932934b911f3 P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:14:58.195765 30924 ts_tablet_manager.cc:1367] T eb065120d1304a2a81ce932934b911f3 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.014s
I20240502 14:14:58.196926 30924 raft_consensus.cc:348] T eb065120d1304a2a81ce932934b911f3 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:58.197335 30924 raft_consensus.cc:374] T eb065120d1304a2a81ce932934b911f3 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:58.197485 30924 raft_consensus.cc:729] T eb065120d1304a2a81ce932934b911f3 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:14:58.197825 30924 consensus_queue.cc:260] T eb065120d1304a2a81ce932934b911f3 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:58.198807 30924 ts_tablet_manager.cc:1398] T eb065120d1304a2a81ce932934b911f3 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:58.199136 26285 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:14:58.199347 26285 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:14:58.203161 26285 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet 14f92995b2e0417687abdc8b35685ef5
I20240502 14:14:58.204178 26538 tablet_service.cc:1514] Processing DeleteTablet for tablet 14f92995b2e0417687abdc8b35685ef5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:58 UTC) from {username='slave'} at 127.0.0.1:51696
I20240502 14:14:58.204587 30931 tablet_bootstrap.cc:492] T 33a3639c26bf4b35a6e15d6ef173751c P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:14:58.204771 30931 ts_tablet_manager.cc:1367] T 33a3639c26bf4b35a6e15d6ef173751c P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.000s
I20240502 14:14:58.205255 29553 tablet_service.cc:1514] Processing DeleteTablet for tablet 14f92995b2e0417687abdc8b35685ef5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:58 UTC) from {username='slave'} at 127.0.0.1:36398
I20240502 14:14:58.205929 30931 raft_consensus.cc:348] T 33a3639c26bf4b35a6e15d6ef173751c P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:58.206213 30931 raft_consensus.cc:374] T 33a3639c26bf4b35a6e15d6ef173751c P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:58.206318 30931 raft_consensus.cc:729] T 33a3639c26bf4b35a6e15d6ef173751c P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:14:58.206655 30931 consensus_queue.cc:260] T 33a3639c26bf4b35a6e15d6ef173751c P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:58.207446 30931 ts_tablet_manager.cc:1398] T 33a3639c26bf4b35a6e15d6ef173751c P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:58.208192 26668 tablet_service.cc:1514] Processing DeleteTablet for tablet 14f92995b2e0417687abdc8b35685ef5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:58 UTC) from {username='slave'} at 127.0.0.1:34150
I20240502 14:14:58.229830 26558 raft_consensus.cc:1203] T 177a34c96aa84cb4aa50bbd79cab4144 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:14:58.235852 30944 tablet_replica.cc:330] T 14f92995b2e0417687abdc8b35685ef5 P 4092902b3e674857a0147f1b4f2cdb86: stopping tablet replica
I20240502 14:14:58.236114 30944 raft_consensus.cc:2229] T 14f92995b2e0417687abdc8b35685ef5 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:58.236351 30944 raft_consensus.cc:2258] T 14f92995b2e0417687abdc8b35685ef5 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:58.237730 30944 ts_tablet_manager.cc:1875] T 14f92995b2e0417687abdc8b35685ef5 P 4092902b3e674857a0147f1b4f2cdb86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:58.241036 30943 tablet_replica.cc:330] T 14f92995b2e0417687abdc8b35685ef5 P ba187e38a2754d60a12de9ac4856430a: stopping tablet replica
I20240502 14:14:58.241274 30943 raft_consensus.cc:2229] T 14f92995b2e0417687abdc8b35685ef5 P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:58.241677 30943 raft_consensus.cc:2258] T 14f92995b2e0417687abdc8b35685ef5 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:58.244220 30943 ts_tablet_manager.cc:1875] T 14f92995b2e0417687abdc8b35685ef5 P ba187e38a2754d60a12de9ac4856430a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:58.250635 30943 ts_tablet_manager.cc:1888] T 14f92995b2e0417687abdc8b35685ef5 P ba187e38a2754d60a12de9ac4856430a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:58.252734 30515 raft_consensus.cc:482] T 929a85cbe95746b798a7510b3a1cc19e P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:58.252898 30515 raft_consensus.cc:504] T 929a85cbe95746b798a7510b3a1cc19e P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:58.253538 30515 leader_election.cc:288] T 929a85cbe95746b798a7510b3a1cc19e P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:14:58.254676 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929a85cbe95746b798a7510b3a1cc19e" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:14:58.254918 29309 raft_consensus.cc:2454] T 929a85cbe95746b798a7510b3a1cc19e P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 0.
I20240502 14:14:58.255316 29111 leader_election.cc:302] T 929a85cbe95746b798a7510b3a1cc19e P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:14:58.255607 30274 raft_consensus.cc:2781] T 929a85cbe95746b798a7510b3a1cc19e P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:58.255730 30274 raft_consensus.cc:482] T 929a85cbe95746b798a7510b3a1cc19e P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:58.255802 30274 raft_consensus.cc:3037] T 929a85cbe95746b798a7510b3a1cc19e P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:58.258965 30943 log.cc:1196] T 14f92995b2e0417687abdc8b35685ef5 P ba187e38a2754d60a12de9ac4856430a: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/14f92995b2e0417687abdc8b35685ef5
I20240502 14:14:58.259634 30943 ts_tablet_manager.cc:1909] T 14f92995b2e0417687abdc8b35685ef5 P ba187e38a2754d60a12de9ac4856430a: Deleting consensus metadata
I20240502 14:14:58.261925 26271 catalog_manager.cc:4830] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): tablet 14f92995b2e0417687abdc8b35685ef5 (table rebalancer_dynamic_table_new_name [id=b7b2caa9d8a84820b3e17aa9f5484bd9]) successfully deleted
I20240502 14:14:58.264207 30274 raft_consensus.cc:504] T 929a85cbe95746b798a7510b3a1cc19e P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:58.265606 30944 ts_tablet_manager.cc:1888] T 14f92995b2e0417687abdc8b35685ef5 P 4092902b3e674857a0147f1b4f2cdb86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:58.266763 30944 log.cc:1196] T 14f92995b2e0417687abdc8b35685ef5 P 4092902b3e674857a0147f1b4f2cdb86: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/wal/wals/14f92995b2e0417687abdc8b35685ef5
I20240502 14:14:58.267158 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929a85cbe95746b798a7510b3a1cc19e" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
I20240502 14:14:58.267418 29309 raft_consensus.cc:3037] T 929a85cbe95746b798a7510b3a1cc19e P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:58.267421 30944 ts_tablet_manager.cc:1909] T 14f92995b2e0417687abdc8b35685ef5 P 4092902b3e674857a0147f1b4f2cdb86: Deleting consensus metadata
I20240502 14:14:58.268183 30945 tablet_replica.cc:330] T 14f92995b2e0417687abdc8b35685ef5 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: stopping tablet replica
I20240502 14:14:58.268563 30945 raft_consensus.cc:2229] T 14f92995b2e0417687abdc8b35685ef5 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:58.268935 30945 raft_consensus.cc:2258] T 14f92995b2e0417687abdc8b35685ef5 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:58.268890 26272 catalog_manager.cc:4830] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097): tablet 14f92995b2e0417687abdc8b35685ef5 (table rebalancer_dynamic_table_new_name [id=b7b2caa9d8a84820b3e17aa9f5484bd9]) successfully deleted
I20240502 14:14:58.270242 29309 raft_consensus.cc:2454] T 929a85cbe95746b798a7510b3a1cc19e P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 1.
I20240502 14:14:58.271178 29111 leader_election.cc:302] T 929a85cbe95746b798a7510b3a1cc19e P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:14:58.271517 30274 leader_election.cc:288] T 929a85cbe95746b798a7510b3a1cc19e P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 election: Requested vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:14:58.272485 30584 raft_consensus.cc:482] T 33a3639c26bf4b35a6e15d6ef173751c P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:58.272640 30584 raft_consensus.cc:504] T 33a3639c26bf4b35a6e15d6ef173751c P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:58.273520 30584 leader_election.cc:288] T 33a3639c26bf4b35a6e15d6ef173751c P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:14:58.274924 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a3639c26bf4b35a6e15d6ef173751c" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:14:58.275143 29443 raft_consensus.cc:2454] T 33a3639c26bf4b35a6e15d6ef173751c P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 0.
I20240502 14:14:58.275892 29245 leader_election.cc:302] T 33a3639c26bf4b35a6e15d6ef173751c P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:14:58.276495 29573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a3639c26bf4b35a6e15d6ef173751c" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86" is_pre_election: true
I20240502 14:14:58.276820 29573 raft_consensus.cc:2454] T 33a3639c26bf4b35a6e15d6ef173751c P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 0.
I20240502 14:14:58.277004 30945 ts_tablet_manager.cc:1875] T 14f92995b2e0417687abdc8b35685ef5 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:58.279218 26819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929a85cbe95746b798a7510b3a1cc19e" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:14:58.279791 26819 raft_consensus.cc:2454] T 929a85cbe95746b798a7510b3a1cc19e P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 0.
I20240502 14:14:58.281448 30584 raft_consensus.cc:2781] T 33a3639c26bf4b35a6e15d6ef173751c P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:58.281594 30584 raft_consensus.cc:482] T 33a3639c26bf4b35a6e15d6ef173751c P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:58.281678 30584 raft_consensus.cc:3037] T 33a3639c26bf4b35a6e15d6ef173751c P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:58.282641 26819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929a85cbe95746b798a7510b3a1cc19e" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:14:58.285805 26819 raft_consensus.cc:3037] T 929a85cbe95746b798a7510b3a1cc19e P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:58.288040 30584 raft_consensus.cc:504] T 33a3639c26bf4b35a6e15d6ef173751c P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:58.288491 30274 raft_consensus.cc:2781] T 929a85cbe95746b798a7510b3a1cc19e P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:58.289079 30274 raft_consensus.cc:686] T 929a85cbe95746b798a7510b3a1cc19e P 1cb67158133e4648a6eceba32d4dbba3 [term 1 LEADER]: Becoming Leader. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Running, Role: LEADER
I20240502 14:14:58.290278 30584 leader_election.cc:288] T 33a3639c26bf4b35a6e15d6ef173751c P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 election: Requested vote from peers 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:14:58.289530 30274 consensus_queue.cc:237] T 929a85cbe95746b798a7510b3a1cc19e P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:58.291342 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a3639c26bf4b35a6e15d6ef173751c" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
I20240502 14:14:58.291625 29443 raft_consensus.cc:3037] T 33a3639c26bf4b35a6e15d6ef173751c P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:58.292883 29573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a3639c26bf4b35a6e15d6ef173751c" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86"
I20240502 14:14:58.293134 29573 raft_consensus.cc:3037] T 33a3639c26bf4b35a6e15d6ef173751c P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:58.294266 29443 raft_consensus.cc:2454] T 33a3639c26bf4b35a6e15d6ef173751c P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 1.
I20240502 14:14:58.294646 26819 raft_consensus.cc:2454] T 929a85cbe95746b798a7510b3a1cc19e P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 1.
I20240502 14:14:58.295940 29573 raft_consensus.cc:2454] T 33a3639c26bf4b35a6e15d6ef173751c P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 1.
I20240502 14:14:58.298277 29245 leader_election.cc:302] T 33a3639c26bf4b35a6e15d6ef173751c P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:14:58.300192 30584 raft_consensus.cc:2781] T 33a3639c26bf4b35a6e15d6ef173751c P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:58.301872 30813 raft_consensus.cc:482] T eb065120d1304a2a81ce932934b911f3 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:58.302085 30813 raft_consensus.cc:504] T eb065120d1304a2a81ce932934b911f3 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:58.302985 30813 leader_election.cc:288] T eb065120d1304a2a81ce932934b911f3 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:14:58.304010 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb065120d1304a2a81ce932934b911f3" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:14:58.304240 29309 raft_consensus.cc:2454] T eb065120d1304a2a81ce932934b911f3 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 0.
I20240502 14:14:58.304850 29174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb065120d1304a2a81ce932934b911f3" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:14:58.305095 29174 raft_consensus.cc:2454] T eb065120d1304a2a81ce932934b911f3 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 0.
I20240502 14:14:58.305908 29378 leader_election.cc:302] T eb065120d1304a2a81ce932934b911f3 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 68a3c2b17ed94e06aba66ed4178132ab; no voters:
I20240502 14:14:58.306488 30813 raft_consensus.cc:2781] T eb065120d1304a2a81ce932934b911f3 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:58.306608 30813 raft_consensus.cc:482] T eb065120d1304a2a81ce932934b911f3 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:58.306694 30813 raft_consensus.cc:3037] T eb065120d1304a2a81ce932934b911f3 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:58.308759 26285 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:14:58.309471 30813 raft_consensus.cc:504] T eb065120d1304a2a81ce932934b911f3 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:58.311745 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb065120d1304a2a81ce932934b911f3" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
I20240502 14:14:58.311998 29309 raft_consensus.cc:3037] T eb065120d1304a2a81ce932934b911f3 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:58.315709 29309 raft_consensus.cc:2454] T eb065120d1304a2a81ce932934b911f3 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 1.
I20240502 14:14:58.316417 29378 leader_election.cc:302] T eb065120d1304a2a81ce932934b911f3 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:14:58.316788 30802 raft_consensus.cc:2781] T eb065120d1304a2a81ce932934b911f3 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:58.319806 30813 leader_election.cc:288] T eb065120d1304a2a81ce932934b911f3 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 election: Requested vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:14:58.320626 29174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb065120d1304a2a81ce932934b911f3" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:14:58.320859 29174 raft_consensus.cc:3037] T eb065120d1304a2a81ce932934b911f3 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:58.323900 29174 raft_consensus.cc:2454] T eb065120d1304a2a81ce932934b911f3 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 1.
I20240502 14:14:58.324761 29553 tablet_service.cc:1467] Processing CreateTablet for tablet b41fb969347644adb57a29ebdb2e6301 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=36a7544a69604c28ae0b948c3186d44a]), partition=
I20240502 14:14:58.325162 29553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b41fb969347644adb57a29ebdb2e6301. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:58.329344 29289 tablet_service.cc:1467] Processing CreateTablet for tablet b41fb969347644adb57a29ebdb2e6301 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=36a7544a69604c28ae0b948c3186d44a]), partition=
I20240502 14:14:58.329617 30945 ts_tablet_manager.cc:1888] T 14f92995b2e0417687abdc8b35685ef5 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:58.329711 29289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b41fb969347644adb57a29ebdb2e6301. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:58.329746 30945 log.cc:1196] T 14f92995b2e0417687abdc8b35685ef5 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/wals/14f92995b2e0417687abdc8b35685ef5
I20240502 14:14:58.332759 29423 tablet_service.cc:1467] Processing CreateTablet for tablet b41fb969347644adb57a29ebdb2e6301 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=36a7544a69604c28ae0b948c3186d44a]), partition=
I20240502 14:14:58.333554 29423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b41fb969347644adb57a29ebdb2e6301. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:58.333146 30802 raft_consensus.cc:686] T eb065120d1304a2a81ce932934b911f3 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 LEADER]: Becoming Leader. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Running, Role: LEADER
I20240502 14:14:58.336792 30802 consensus_queue.cc:237] T eb065120d1304a2a81ce932934b911f3 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:58.338799 30945 ts_tablet_manager.cc:1909] T 14f92995b2e0417687abdc8b35685ef5 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting consensus metadata
I20240502 14:14:58.341056 26272 catalog_manager.cc:4830] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): tablet 14f92995b2e0417687abdc8b35685ef5 (table rebalancer_dynamic_table_new_name [id=b7b2caa9d8a84820b3e17aa9f5484bd9]) successfully deleted
I20240502 14:14:58.341995 30925 tablet_bootstrap.cc:492] T b41fb969347644adb57a29ebdb2e6301 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:14:58.343628 30925 tablet_bootstrap.cc:654] T b41fb969347644adb57a29ebdb2e6301 P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:58.343976 30924 tablet_bootstrap.cc:492] T b41fb969347644adb57a29ebdb2e6301 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:14:58.346815 30924 tablet_bootstrap.cc:654] T b41fb969347644adb57a29ebdb2e6301 P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:58.354902 30931 tablet_bootstrap.cc:492] T b41fb969347644adb57a29ebdb2e6301 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:14:58.356726 30931 tablet_bootstrap.cc:654] T b41fb969347644adb57a29ebdb2e6301 P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:58.388597 30584 raft_consensus.cc:686] T 33a3639c26bf4b35a6e15d6ef173751c P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 LEADER]: Becoming Leader. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Running, Role: LEADER
I20240502 14:14:58.389045 30584 consensus_queue.cc:237] T 33a3639c26bf4b35a6e15d6ef173751c P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:14:58.389932 30925 tablet_bootstrap.cc:492] T b41fb969347644adb57a29ebdb2e6301 P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:14:58.391252 30931 tablet_bootstrap.cc:492] T b41fb969347644adb57a29ebdb2e6301 P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:14:58.391736 30925 ts_tablet_manager.cc:1367] T b41fb969347644adb57a29ebdb2e6301 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.050s user 0.007s sys 0.005s
I20240502 14:14:58.392206 30931 ts_tablet_manager.cc:1367] T b41fb969347644adb57a29ebdb2e6301 P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.010s
I20240502 14:14:58.393193 30925 raft_consensus.cc:348] T b41fb969347644adb57a29ebdb2e6301 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:58.394249 30931 raft_consensus.cc:348] T b41fb969347644adb57a29ebdb2e6301 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:58.394567 30931 raft_consensus.cc:374] T b41fb969347644adb57a29ebdb2e6301 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:58.394716 30925 raft_consensus.cc:374] T b41fb969347644adb57a29ebdb2e6301 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:58.394876 30931 raft_consensus.cc:729] T b41fb969347644adb57a29ebdb2e6301 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:14:58.395066 30925 raft_consensus.cc:729] T b41fb969347644adb57a29ebdb2e6301 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:14:58.397661 30924 tablet_bootstrap.cc:492] T b41fb969347644adb57a29ebdb2e6301 P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:14:58.398115 30924 ts_tablet_manager.cc:1367] T b41fb969347644adb57a29ebdb2e6301 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.054s user 0.007s sys 0.003s
I20240502 14:14:58.398103 30931 consensus_queue.cc:260] T b41fb969347644adb57a29ebdb2e6301 P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:58.398615 30925 consensus_queue.cc:260] T b41fb969347644adb57a29ebdb2e6301 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:58.398989 30924 raft_consensus.cc:348] T b41fb969347644adb57a29ebdb2e6301 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:58.401422 26285 catalog_manager.cc:5484] T 33a3639c26bf4b35a6e15d6ef173751c P c1100e16b3b9495ca61d4f95e89b6a89 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198). New cstate: current_term: 1 leader_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:14:58.399876 30924 raft_consensus.cc:374] T b41fb969347644adb57a29ebdb2e6301 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:58.404242 30924 raft_consensus.cc:729] T b41fb969347644adb57a29ebdb2e6301 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:14:58.405732 30931 ts_tablet_manager.cc:1398] T b41fb969347644adb57a29ebdb2e6301 P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20240502 14:14:58.407115 26280 catalog_manager.cc:5484] T eb065120d1304a2a81ce932934b911f3 P 68a3c2b17ed94e06aba66ed4178132ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199). New cstate: current_term: 1 leader_uuid: "68a3c2b17ed94e06aba66ed4178132ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:14:58.408212 30925 ts_tablet_manager.cc:1398] T b41fb969347644adb57a29ebdb2e6301 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:14:58.408610 30924 consensus_queue.cc:260] T b41fb969347644adb57a29ebdb2e6301 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:58.408905 26279 catalog_manager.cc:5484] T 929a85cbe95746b798a7510b3a1cc19e P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197). New cstate: current_term: 1 leader_uuid: "1cb67158133e4648a6eceba32d4dbba3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:14:58.409291 30924 ts_tablet_manager.cc:1398] T b41fb969347644adb57a29ebdb2e6301 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240502 14:14:58.469156 26558 consensus_queue.cc:237] T 62218b2a597e4fb188ced6d2b9a83a59 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:58.471242 29573 raft_consensus.cc:1261] T 62218b2a597e4fb188ced6d2b9a83a59 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:58.472222 26689 raft_consensus.cc:1261] T 62218b2a597e4fb188ced6d2b9a83a59 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:58.472322 30926 consensus_queue.cc:1035] T 62218b2a597e4fb188ced6d2b9a83a59 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:58.475543 30615 consensus_queue.cc:1035] T 62218b2a597e4fb188ced6d2b9a83a59 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:58.476244 30470 raft_consensus.cc:2932] T 62218b2a597e4fb188ced6d2b9a83a59 P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:14:58.476920 29573 raft_consensus.cc:2932] T 62218b2a597e4fb188ced6d2b9a83a59 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:14:58.477999 26689 raft_consensus.cc:2932] T 62218b2a597e4fb188ced6d2b9a83a59 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:14:58.482865 26280 catalog_manager.cc:5484] T 62218b2a597e4fb188ced6d2b9a83a59 P ba187e38a2754d60a12de9ac4856430a reported cstate change: config changed from index 4 to 5, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:14:58.487089 26797 tablet_service.cc:1514] Processing DeleteTablet for tablet 62218b2a597e4fb188ced6d2b9a83a59 with delete_type TABLET_DATA_TOMBSTONED (TS e4cfb8fc5a474177af6848a6efa13a27 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:56010
I20240502 14:14:58.488094 30974 tablet_replica.cc:330] T 62218b2a597e4fb188ced6d2b9a83a59 P e4cfb8fc5a474177af6848a6efa13a27: stopping tablet replica
I20240502 14:14:58.488308 30974 raft_consensus.cc:2229] T 62218b2a597e4fb188ced6d2b9a83a59 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:58.488662 30974 raft_consensus.cc:2258] T 62218b2a597e4fb188ced6d2b9a83a59 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:58.489490 30974 ts_tablet_manager.cc:1875] T 62218b2a597e4fb188ced6d2b9a83a59 P e4cfb8fc5a474177af6848a6efa13a27: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:14:58.497164 30974 ts_tablet_manager.cc:1888] T 62218b2a597e4fb188ced6d2b9a83a59 P e4cfb8fc5a474177af6848a6efa13a27: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:14:58.497400 30974 log.cc:1196] T 62218b2a597e4fb188ced6d2b9a83a59 P e4cfb8fc5a474177af6848a6efa13a27: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/wals/62218b2a597e4fb188ced6d2b9a83a59
I20240502 14:14:58.498438 26272 catalog_manager.cc:4830] TS e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): tablet 62218b2a597e4fb188ced6d2b9a83a59 (table rebalance_test_table_15 [id=ff3107a4d4c94012a52f0cd0ca9ef235]) successfully deleted
I20240502 14:14:58.671042 30584 raft_consensus.cc:482] T b41fb969347644adb57a29ebdb2e6301 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:58.671254 30584 raft_consensus.cc:504] T b41fb969347644adb57a29ebdb2e6301 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:58.671926 30584 leader_election.cc:288] T b41fb969347644adb57a29ebdb2e6301 P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:14:58.672349 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b41fb969347644adb57a29ebdb2e6301" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:14:58.672536 29573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b41fb969347644adb57a29ebdb2e6301" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86" is_pre_election: true
I20240502 14:14:58.672612 29443 raft_consensus.cc:2454] T b41fb969347644adb57a29ebdb2e6301 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 0.
I20240502 14:14:58.672798 29573 raft_consensus.cc:2454] T b41fb969347644adb57a29ebdb2e6301 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 0.
I20240502 14:14:58.673039 29245 leader_election.cc:302] T b41fb969347644adb57a29ebdb2e6301 P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:14:58.673353 30584 raft_consensus.cc:2781] T b41fb969347644adb57a29ebdb2e6301 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:58.673466 30584 raft_consensus.cc:482] T b41fb969347644adb57a29ebdb2e6301 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:58.673580 30584 raft_consensus.cc:3037] T b41fb969347644adb57a29ebdb2e6301 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:58.675668 30584 raft_consensus.cc:504] T b41fb969347644adb57a29ebdb2e6301 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:58.676285 30584 leader_election.cc:288] T b41fb969347644adb57a29ebdb2e6301 P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 election: Requested vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:14:58.676770 29573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b41fb969347644adb57a29ebdb2e6301" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86"
I20240502 14:14:58.676756 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b41fb969347644adb57a29ebdb2e6301" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
I20240502 14:14:58.676985 29443 raft_consensus.cc:3037] T b41fb969347644adb57a29ebdb2e6301 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:58.676973 29573 raft_consensus.cc:3037] T b41fb969347644adb57a29ebdb2e6301 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:58.679239 29573 raft_consensus.cc:2454] T b41fb969347644adb57a29ebdb2e6301 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 1.
I20240502 14:14:58.679298 29443 raft_consensus.cc:2454] T b41fb969347644adb57a29ebdb2e6301 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 1.
I20240502 14:14:58.679971 29245 leader_election.cc:302] T b41fb969347644adb57a29ebdb2e6301 P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:14:58.680223 30584 raft_consensus.cc:2781] T b41fb969347644adb57a29ebdb2e6301 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:58.680411 30584 raft_consensus.cc:686] T b41fb969347644adb57a29ebdb2e6301 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 LEADER]: Becoming Leader. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Running, Role: LEADER
I20240502 14:14:58.680709 30584 consensus_queue.cc:237] T b41fb969347644adb57a29ebdb2e6301 P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:14:58.683959 26276 catalog_manager.cc:5484] T b41fb969347644adb57a29ebdb2e6301 P c1100e16b3b9495ca61d4f95e89b6a89 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198). New cstate: current_term: 1 leader_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:14:58.707613 30274 consensus_queue.cc:1035] T 929a85cbe95746b798a7510b3a1cc19e P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:58.712759 29156 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:14:58.748745 29423 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:14:58.753629 29553 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:14:58.762003 30953 consensus_queue.cc:1035] T 929a85cbe95746b798a7510b3a1cc19e P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:58.771952 26668 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:14:58.777849 26538 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:14:58.807927 30584 consensus_queue.cc:1035] T 33a3639c26bf4b35a6e15d6ef173751c P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:58.811983 26406 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:14:58.836421 30947 consensus_queue.cc:1035] T 33a3639c26bf4b35a6e15d6ef173751c P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:58.839913 29289 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:14:58.841403 26276 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:14:58.851032 30802 consensus_queue.cc:1035] T eb065120d1304a2a81ce932934b911f3 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:58.861514 29575 raft_consensus.cc:1261] T b41fb969347644adb57a29ebdb2e6301 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Refusing update from remote peer c1100e16b3b9495ca61d4f95e89b6a89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:58.862459 29442 raft_consensus.cc:1261] T b41fb969347644adb57a29ebdb2e6301 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer c1100e16b3b9495ca61d4f95e89b6a89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:58.862911 30947 consensus_queue.cc:1035] T b41fb969347644adb57a29ebdb2e6301 P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:58.863452 30584 consensus_queue.cc:1035] T b41fb969347644adb57a29ebdb2e6301 P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:58.924031 30802 consensus_queue.cc:1035] T eb065120d1304a2a81ce932934b911f3 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:58.932358 26280 catalog_manager.cc:5576] TS 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781) does not have the latest schema for tablet b41fb969347644adb57a29ebdb2e6301 (table rebalancer_dynamic_table [id=36a7544a69604c28ae0b948c3186d44a]). Expected version 1 got 0
I20240502 14:14:58.942179 31052 tablet.cc:1722] T b41fb969347644adb57a29ebdb2e6301 P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:58.949111 31053 tablet.cc:1722] T b41fb969347644adb57a29ebdb2e6301 P 68a3c2b17ed94e06aba66ed4178132ab: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:58.949718 31053 tablet.cc:1620] T b41fb969347644adb57a29ebdb2e6301 P 68a3c2b17ed94e06aba66ed4178132ab: MemRowSet was empty: no flush needed.
I20240502 14:14:58.952273 26280 catalog_manager.cc:5576] TS 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781) does not have the latest schema for tablet b41fb969347644adb57a29ebdb2e6301 (table rebalancer_dynamic_table [id=36a7544a69604c28ae0b948c3186d44a]). Expected version 1 got 0
I20240502 14:14:58.953755 26797 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:14:58.954475 31052 tablet.cc:1620] T b41fb969347644adb57a29ebdb2e6301 P c1100e16b3b9495ca61d4f95e89b6a89: MemRowSet was empty: no flush needed.
I20240502 14:14:58.952896 26276 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_extra_table_20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:58.975667 29423 tablet_service.cc:1467] Processing CreateTablet for tablet 47e87174282541acb60a41a32766dee4 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=2ccf4597fbc1491f87f24cd5733a46d2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:58.976127 26797 tablet_service.cc:1467] Processing CreateTablet for tablet 47e87174282541acb60a41a32766dee4 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=2ccf4597fbc1491f87f24cd5733a46d2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:58.976230 29423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47e87174282541acb60a41a32766dee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:58.976644 26797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47e87174282541acb60a41a32766dee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:58.979597 29553 tablet_service.cc:1467] Processing CreateTablet for tablet fc6401f0ab8c485399325df7c8dd96ad (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=2ccf4597fbc1491f87f24cd5733a46d2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:58.980031 29553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc6401f0ab8c485399325df7c8dd96ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:58.981046 29156 tablet_service.cc:1467] Processing CreateTablet for tablet 47e87174282541acb60a41a32766dee4 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=2ccf4597fbc1491f87f24cd5733a46d2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:58.981588 29156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47e87174282541acb60a41a32766dee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:58.985035 29422 tablet_service.cc:1467] Processing CreateTablet for tablet fc6401f0ab8c485399325df7c8dd96ad (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=2ccf4597fbc1491f87f24cd5733a46d2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:58.985463 29422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc6401f0ab8c485399325df7c8dd96ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:58.993330 26538 tablet_service.cc:1467] Processing CreateTablet for tablet fc6401f0ab8c485399325df7c8dd96ad (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=2ccf4597fbc1491f87f24cd5733a46d2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:58.993719 26538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc6401f0ab8c485399325df7c8dd96ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:59.008181 31056 tablet_bootstrap.cc:492] T fc6401f0ab8c485399325df7c8dd96ad P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:14:59.014961 29421 tablet_service.cc:1467] Processing CreateTablet for tablet e1bdebd770244537a0119baa4e3f8091 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=2ccf4597fbc1491f87f24cd5733a46d2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:59.015789 26406 tablet_service.cc:1467] Processing CreateTablet for tablet e1bdebd770244537a0119baa4e3f8091 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=2ccf4597fbc1491f87f24cd5733a46d2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:59.016194 26279 catalog_manager.cc:5576] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097) does not have the latest schema for tablet b41fb969347644adb57a29ebdb2e6301 (table rebalancer_dynamic_table [id=36a7544a69604c28ae0b948c3186d44a]). Expected version 1 got 0
I20240502 14:14:59.016335 26406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1bdebd770244537a0119baa4e3f8091. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:59.021822 29154 tablet_service.cc:1467] Processing CreateTablet for tablet e1bdebd770244537a0119baa4e3f8091 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=2ccf4597fbc1491f87f24cd5733a46d2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:59.022511 29154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1bdebd770244537a0119baa4e3f8091. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:59.029165 26283 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=36a7544a69604c28ae0b948c3186d44a] alter complete (version 1)
I20240502 14:14:59.033424 31058 tablet_bootstrap.cc:492] T fc6401f0ab8c485399325df7c8dd96ad P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:59.037932 31058 tablet_bootstrap.cc:654] T fc6401f0ab8c485399325df7c8dd96ad P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:59.039996 31060 tablet_bootstrap.cc:492] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:14:59.044126 29421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1bdebd770244537a0119baa4e3f8091. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:59.050560 31059 tablet_bootstrap.cc:492] T fc6401f0ab8c485399325df7c8dd96ad P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:14:59.056339 31060 tablet_bootstrap.cc:654] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:59.056998 31056 tablet_bootstrap.cc:654] T fc6401f0ab8c485399325df7c8dd96ad P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:59.058342 31059 tablet_bootstrap.cc:654] T fc6401f0ab8c485399325df7c8dd96ad P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:59.060005 31058 tablet_bootstrap.cc:492] T fc6401f0ab8c485399325df7c8dd96ad P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:59.060313 31058 ts_tablet_manager.cc:1367] T fc6401f0ab8c485399325df7c8dd96ad P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.000s
I20240502 14:14:59.062466 31058 raft_consensus.cc:348] T fc6401f0ab8c485399325df7c8dd96ad P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:59.063089 31058 raft_consensus.cc:374] T fc6401f0ab8c485399325df7c8dd96ad P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:59.063279 31058 raft_consensus.cc:729] T fc6401f0ab8c485399325df7c8dd96ad P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:59.063606 31058 consensus_queue.cc:260] T fc6401f0ab8c485399325df7c8dd96ad P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:59.064689 30926 raft_consensus.cc:482] T fc6401f0ab8c485399325df7c8dd96ad P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:59.064910 30926 raft_consensus.cc:504] T fc6401f0ab8c485399325df7c8dd96ad P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:59.065683 30926 leader_election.cc:288] T fc6401f0ab8c485399325df7c8dd96ad P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:14:59.066128 31057 tablet_bootstrap.cc:492] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:14:59.067539 26283 catalog_manager.cc:5576] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097) does not have the latest schema for tablet b41fb969347644adb57a29ebdb2e6301 (table rebalancer_dynamic_table [id=36a7544a69604c28ae0b948c3186d44a]). Expected version 1 got 0
I20240502 14:14:59.069643 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc6401f0ab8c485399325df7c8dd96ad" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
W20240502 14:14:59.071414 26494 leader_election.cc:341] T fc6401f0ab8c485399325df7c8dd96ad P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:59.072742 31058 ts_tablet_manager.cc:1398] T fc6401f0ab8c485399325df7c8dd96ad P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240502 14:14:59.076633 31060 tablet_bootstrap.cc:492] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:14:59.076912 31060 ts_tablet_manager.cc:1367] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.002s
I20240502 14:14:59.078220 31060 raft_consensus.cc:348] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:59.078521 31060 raft_consensus.cc:374] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:59.078680 31060 raft_consensus.cc:729] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:14:59.062932 31054 tablet.cc:1722] T b41fb969347644adb57a29ebdb2e6301 P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:59.069362 29575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc6401f0ab8c485399325df7c8dd96ad" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86" is_pre_election: true
I20240502 14:14:59.079322 31060 consensus_queue.cc:260] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:59.079946 31054 tablet.cc:1620] T b41fb969347644adb57a29ebdb2e6301 P 4092902b3e674857a0147f1b4f2cdb86: MemRowSet was empty: no flush needed.
W20240502 14:14:59.079947 26494 leader_election.cc:341] T fc6401f0ab8c485399325df7c8dd96ad P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:59.080103 26494 leader_election.cc:302] T fc6401f0ab8c485399325df7c8dd96ad P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 4092902b3e674857a0147f1b4f2cdb86, 68a3c2b17ed94e06aba66ed4178132ab
I20240502 14:14:59.080581 30926 raft_consensus.cc:2726] T fc6401f0ab8c485399325df7c8dd96ad P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:14:59.084370 31060 ts_tablet_manager.cc:1398] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:14:59.070178 31057 tablet_bootstrap.cc:654] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:59.102468 31055 tablet_bootstrap.cc:492] T 47e87174282541acb60a41a32766dee4 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:14:59.105113 31055 tablet_bootstrap.cc:654] T 47e87174282541acb60a41a32766dee4 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:59.108003 31059 tablet_bootstrap.cc:492] T fc6401f0ab8c485399325df7c8dd96ad P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:14:59.108247 31059 ts_tablet_manager.cc:1367] T fc6401f0ab8c485399325df7c8dd96ad P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.058s user 0.011s sys 0.011s
I20240502 14:14:59.109184 31059 raft_consensus.cc:348] T fc6401f0ab8c485399325df7c8dd96ad P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:59.109414 31059 raft_consensus.cc:374] T fc6401f0ab8c485399325df7c8dd96ad P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:59.111546 31056 tablet_bootstrap.cc:492] T fc6401f0ab8c485399325df7c8dd96ad P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:14:59.111742 31056 ts_tablet_manager.cc:1367] T fc6401f0ab8c485399325df7c8dd96ad P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.104s user 0.005s sys 0.000s
I20240502 14:14:59.112640 31056 raft_consensus.cc:348] T fc6401f0ab8c485399325df7c8dd96ad P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:59.112896 31056 raft_consensus.cc:374] T fc6401f0ab8c485399325df7c8dd96ad P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:59.113004 31056 raft_consensus.cc:729] T fc6401f0ab8c485399325df7c8dd96ad P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:14:59.113268 31056 consensus_queue.cc:260] T fc6401f0ab8c485399325df7c8dd96ad P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:59.113896 31056 ts_tablet_manager.cc:1398] T fc6401f0ab8c485399325df7c8dd96ad P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:59.113976 31059 raft_consensus.cc:729] T fc6401f0ab8c485399325df7c8dd96ad P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:14:59.114328 31056 tablet_bootstrap.cc:492] T 47e87174282541acb60a41a32766dee4 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:14:59.114337 31059 consensus_queue.cc:260] T fc6401f0ab8c485399325df7c8dd96ad P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:59.115018 31059 ts_tablet_manager.cc:1398] T fc6401f0ab8c485399325df7c8dd96ad P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 14:14:59.122591 31057 tablet_bootstrap.cc:492] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:14:59.125857 31057 ts_tablet_manager.cc:1367] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.060s user 0.005s sys 0.000s
I20240502 14:14:59.127112 31057 raft_consensus.cc:348] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:59.117525 31056 tablet_bootstrap.cc:654] T 47e87174282541acb60a41a32766dee4 P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:59.130791 31057 raft_consensus.cc:374] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:59.130968 31057 raft_consensus.cc:729] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:14:59.131278 31057 consensus_queue.cc:260] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:59.132481 31056 tablet_bootstrap.cc:492] T 47e87174282541acb60a41a32766dee4 P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:14:59.132336 26280 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:14:59.136359 31057 ts_tablet_manager.cc:1398] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240502 14:14:59.143913 31052 tablet.cc:1722] T b41fb969347644adb57a29ebdb2e6301 P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:59.144542 31052 tablet.cc:1620] T b41fb969347644adb57a29ebdb2e6301 P c1100e16b3b9495ca61d4f95e89b6a89: MemRowSet was empty: no flush needed.
I20240502 14:14:59.147125 31053 tablet.cc:1722] T b41fb969347644adb57a29ebdb2e6301 P 68a3c2b17ed94e06aba66ed4178132ab: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:59.147644 31053 tablet.cc:1620] T b41fb969347644adb57a29ebdb2e6301 P 68a3c2b17ed94e06aba66ed4178132ab: MemRowSet was empty: no flush needed.
I20240502 14:14:59.150806 31054 tablet.cc:1722] T b41fb969347644adb57a29ebdb2e6301 P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:59.151347 31054 tablet.cc:1620] T b41fb969347644adb57a29ebdb2e6301 P 4092902b3e674857a0147f1b4f2cdb86: MemRowSet was empty: no flush needed.
I20240502 14:14:59.169955 31056 ts_tablet_manager.cc:1367] T 47e87174282541acb60a41a32766dee4 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.056s user 0.002s sys 0.002s
I20240502 14:14:59.170882 31056 raft_consensus.cc:348] T 47e87174282541acb60a41a32766dee4 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:59.171156 31056 raft_consensus.cc:374] T 47e87174282541acb60a41a32766dee4 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:59.171247 31056 raft_consensus.cc:729] T 47e87174282541acb60a41a32766dee4 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:14:59.171515 31056 consensus_queue.cc:260] T 47e87174282541acb60a41a32766dee4 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:59.172127 31056 ts_tablet_manager.cc:1398] T 47e87174282541acb60a41a32766dee4 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:59.172534 31056 tablet_bootstrap.cc:492] T e1bdebd770244537a0119baa4e3f8091 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:14:59.173117 31057 tablet_bootstrap.cc:492] T e1bdebd770244537a0119baa4e3f8091 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:14:59.174729 31056 tablet_bootstrap.cc:654] T e1bdebd770244537a0119baa4e3f8091 P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:59.179095 31057 tablet_bootstrap.cc:654] T e1bdebd770244537a0119baa4e3f8091 P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:59.179225 26280 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=36a7544a69604c28ae0b948c3186d44a] alter complete (version 2)
I20240502 14:14:59.185664 31057 tablet_bootstrap.cc:492] T e1bdebd770244537a0119baa4e3f8091 P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:14:59.186219 31057 ts_tablet_manager.cc:1367] T e1bdebd770244537a0119baa4e3f8091 P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.007s
I20240502 14:14:59.188161 31055 tablet_bootstrap.cc:492] T 47e87174282541acb60a41a32766dee4 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:14:59.188146 31057 raft_consensus.cc:348] T e1bdebd770244537a0119baa4e3f8091 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:59.188553 31055 ts_tablet_manager.cc:1367] T 47e87174282541acb60a41a32766dee4 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.087s user 0.005s sys 0.000s
I20240502 14:14:59.188621 31057 raft_consensus.cc:374] T e1bdebd770244537a0119baa4e3f8091 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:59.188897 31057 raft_consensus.cc:729] T e1bdebd770244537a0119baa4e3f8091 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:14:59.189225 31057 consensus_queue.cc:260] T e1bdebd770244537a0119baa4e3f8091 P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:59.189437 31055 raft_consensus.cc:348] T 47e87174282541acb60a41a32766dee4 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:59.189653 31055 raft_consensus.cc:374] T 47e87174282541acb60a41a32766dee4 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:59.189749 31055 raft_consensus.cc:729] T 47e87174282541acb60a41a32766dee4 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:14:59.189982 31055 consensus_queue.cc:260] T 47e87174282541acb60a41a32766dee4 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:59.190618 31055 ts_tablet_manager.cc:1398] T 47e87174282541acb60a41a32766dee4 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:14:59.191468 31057 ts_tablet_manager.cc:1398] T e1bdebd770244537a0119baa4e3f8091 P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20240502 14:14:59.192379 31056 tablet_bootstrap.cc:492] T e1bdebd770244537a0119baa4e3f8091 P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:14:59.192683 31056 ts_tablet_manager.cc:1367] T e1bdebd770244537a0119baa4e3f8091 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.007s
I20240502 14:14:59.193682 31056 raft_consensus.cc:348] T e1bdebd770244537a0119baa4e3f8091 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:59.193984 31056 raft_consensus.cc:374] T e1bdebd770244537a0119baa4e3f8091 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:59.194120 31056 raft_consensus.cc:729] T e1bdebd770244537a0119baa4e3f8091 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:14:59.194379 31056 consensus_queue.cc:260] T e1bdebd770244537a0119baa4e3f8091 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:59.194983 31056 ts_tablet_manager.cc:1398] T e1bdebd770244537a0119baa4e3f8091 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:59.271256 30953 raft_consensus.cc:482] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:59.271539 30953 raft_consensus.cc:504] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:59.272580 30953 leader_election.cc:288] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:59.274019 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47e87174282541acb60a41a32766dee4" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:14:59.274361 29443 raft_consensus.cc:2454] T 47e87174282541acb60a41a32766dee4 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 0.
I20240502 14:14:59.275264 26819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47e87174282541acb60a41a32766dee4" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:14:59.275563 26819 raft_consensus.cc:2454] T 47e87174282541acb60a41a32766dee4 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 0.
I20240502 14:14:59.276409 29112 leader_election.cc:302] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 68a3c2b17ed94e06aba66ed4178132ab; no voters:
I20240502 14:14:59.277454 30953 raft_consensus.cc:2781] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:59.277580 30953 raft_consensus.cc:482] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:59.277701 30953 raft_consensus.cc:3037] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:59.280567 30953 raft_consensus.cc:504] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:59.282094 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47e87174282541acb60a41a32766dee4" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
I20240502 14:14:59.282363 29443 raft_consensus.cc:3037] T 47e87174282541acb60a41a32766dee4 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:59.285306 29443 raft_consensus.cc:2454] T 47e87174282541acb60a41a32766dee4 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 1.
I20240502 14:14:59.285816 29112 leader_election.cc:302] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 68a3c2b17ed94e06aba66ed4178132ab; no voters:
I20240502 14:14:59.286165 30274 raft_consensus.cc:2781] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:59.286798 26558 consensus_queue.cc:237] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:14:59.287129 30274 raft_consensus.cc:686] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 LEADER]: Becoming Leader. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Running, Role: LEADER
I20240502 14:14:59.287276 26279 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:14:59.287448 30274 consensus_queue.cc:237] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:59.289351 26689 raft_consensus.cc:1261] T 66554eb8b44145d4b474707b061c9659 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:59.290359 29308 raft_consensus.cc:1261] T 66554eb8b44145d4b474707b061c9659 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:59.291003 30926 consensus_queue.cc:1035] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:59.291373 30953 leader_election.cc:288] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 election: Requested vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:14:59.291409 26819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47e87174282541acb60a41a32766dee4" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:14:59.291623 26819 raft_consensus.cc:3037] T 47e87174282541acb60a41a32766dee4 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:59.292378 30926 consensus_queue.cc:1035] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20240502 14:14:59.295513 26819 raft_consensus.cc:2454] T 47e87174282541acb60a41a32766dee4 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 1.
I20240502 14:14:59.301570 31052 tablet.cc:1722] T b41fb969347644adb57a29ebdb2e6301 P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:59.302551 26279 catalog_manager.cc:5484] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197). New cstate: current_term: 1 leader_uuid: "1cb67158133e4648a6eceba32d4dbba3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:14:59.303390 31053 tablet.cc:1722] T b41fb969347644adb57a29ebdb2e6301 P 68a3c2b17ed94e06aba66ed4178132ab: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:59.307749 30926 raft_consensus.cc:2932] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:14:59.309262 29309 raft_consensus.cc:2932] T 66554eb8b44145d4b474707b061c9659 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:14:59.312484 26689 raft_consensus.cc:2932] T 66554eb8b44145d4b474707b061c9659 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:14:59.327042 30165 raft_consensus.cc:482] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:59.327250 30165 raft_consensus.cc:504] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:59.327891 30165 leader_election.cc:288] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:14:59.328645 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1bdebd770244537a0119baa4e3f8091" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:14:59.328747 31054 tablet.cc:1722] T b41fb969347644adb57a29ebdb2e6301 P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:59.328919 29443 raft_consensus.cc:2454] T e1bdebd770244537a0119baa4e3f8091 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:14:59.328981 29174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1bdebd770244537a0119baa4e3f8091" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:14:59.329281 29174 raft_consensus.cc:2454] T e1bdebd770244537a0119baa4e3f8091 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:14:59.329952 26362 leader_election.cc:302] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:14:59.331475 30165 raft_consensus.cc:2781] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:59.331614 30165 raft_consensus.cc:482] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:59.331894 30165 raft_consensus.cc:3037] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:59.342571 30165 raft_consensus.cc:504] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:59.343763 29174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1bdebd770244537a0119baa4e3f8091" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:14:59.343780 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1bdebd770244537a0119baa4e3f8091" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
I20240502 14:14:59.321931 26280 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table_new_name [id=36a7544a69604c28ae0b948c3186d44a] alter complete (version 3)
I20240502 14:14:59.325511 26284 catalog_manager.cc:5484] T 66554eb8b44145d4b474707b061c9659 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: config changed from index 4 to 5, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) evicted. New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:14:59.343986 29443 raft_consensus.cc:3037] T e1bdebd770244537a0119baa4e3f8091 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:59.344008 29174 raft_consensus.cc:3037] T e1bdebd770244537a0119baa4e3f8091 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:59.347005 29443 raft_consensus.cc:2454] T e1bdebd770244537a0119baa4e3f8091 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:14:59.348785 26363 leader_election.cc:302] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:14:59.349097 30791 raft_consensus.cc:2781] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:59.352001 29174 raft_consensus.cc:2454] T e1bdebd770244537a0119baa4e3f8091 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:14:59.355369 26406 tablet_service.cc:1514] Processing DeleteTablet for tablet 66554eb8b44145d4b474707b061c9659 with delete_type TABLET_DATA_TOMBSTONED (TS 89af7f423cf44717a08d6b142f0ed49d not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:37422
I20240502 14:14:59.343355 30165 leader_election.cc:288] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Requested vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:14:59.361420 31098 tablet_replica.cc:330] T 66554eb8b44145d4b474707b061c9659 P 89af7f423cf44717a08d6b142f0ed49d: stopping tablet replica
I20240502 14:14:59.361727 31098 raft_consensus.cc:2229] T 66554eb8b44145d4b474707b061c9659 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:59.362167 31098 raft_consensus.cc:2258] T 66554eb8b44145d4b474707b061c9659 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:59.361965 30791 raft_consensus.cc:686] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Becoming Leader. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Running, Role: LEADER
I20240502 14:14:59.362869 30791 consensus_queue.cc:237] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:59.363483 31098 ts_tablet_manager.cc:1875] T 66554eb8b44145d4b474707b061c9659 P 89af7f423cf44717a08d6b142f0ed49d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:14:59.373701 26283 catalog_manager.cc:5484] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: term changed from 0 to 1, leader changed from <none> to 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193). New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:14:59.375000 31098 ts_tablet_manager.cc:1888] T 66554eb8b44145d4b474707b061c9659 P 89af7f423cf44717a08d6b142f0ed49d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:14:59.375140 31098 log.cc:1196] T 66554eb8b44145d4b474707b061c9659 P 89af7f423cf44717a08d6b142f0ed49d: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/66554eb8b44145d4b474707b061c9659
I20240502 14:14:59.376086 26271 catalog_manager.cc:4830] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): tablet 66554eb8b44145d4b474707b061c9659 (table rebalance_test_table_3 [id=e814fcdd8ac94d6eb54d1c57f004c870]) successfully deleted
I20240502 14:14:59.440639 30448 raft_consensus.cc:482] T fc6401f0ab8c485399325df7c8dd96ad P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:59.440846 30448 raft_consensus.cc:504] T fc6401f0ab8c485399325df7c8dd96ad P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:59.441691 30448 leader_election.cc:288] T fc6401f0ab8c485399325df7c8dd96ad P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:14:59.442378 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc6401f0ab8c485399325df7c8dd96ad" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:14:59.442636 29443 raft_consensus.cc:2454] T fc6401f0ab8c485399325df7c8dd96ad P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:14:59.443040 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc6401f0ab8c485399325df7c8dd96ad" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:59.443199 29511 leader_election.cc:302] T fc6401f0ab8c485399325df7c8dd96ad P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, 68a3c2b17ed94e06aba66ed4178132ab; no voters:
I20240502 14:14:59.443254 26558 raft_consensus.cc:2454] T fc6401f0ab8c485399325df7c8dd96ad P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:14:59.443807 30448 raft_consensus.cc:2781] T fc6401f0ab8c485399325df7c8dd96ad P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:59.443933 30448 raft_consensus.cc:482] T fc6401f0ab8c485399325df7c8dd96ad P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:59.444027 30448 raft_consensus.cc:3037] T fc6401f0ab8c485399325df7c8dd96ad P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:59.446877 30448 raft_consensus.cc:504] T fc6401f0ab8c485399325df7c8dd96ad P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:59.448026 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc6401f0ab8c485399325df7c8dd96ad" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
I20240502 14:14:59.448275 29443 raft_consensus.cc:3037] T fc6401f0ab8c485399325df7c8dd96ad P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:59.449200 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc6401f0ab8c485399325df7c8dd96ad" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:14:59.449395 26558 raft_consensus.cc:3037] T fc6401f0ab8c485399325df7c8dd96ad P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:59.452260 26558 raft_consensus.cc:2454] T fc6401f0ab8c485399325df7c8dd96ad P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:14:59.452849 29510 leader_election.cc:302] T fc6401f0ab8c485399325df7c8dd96ad P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:59.454320 29443 raft_consensus.cc:2454] T fc6401f0ab8c485399325df7c8dd96ad P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:14:59.455765 30448 leader_election.cc:288] T fc6401f0ab8c485399325df7c8dd96ad P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Requested vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:14:59.455991 30448 raft_consensus.cc:2781] T fc6401f0ab8c485399325df7c8dd96ad P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:59.456099 30448 raft_consensus.cc:686] T fc6401f0ab8c485399325df7c8dd96ad P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Becoming Leader. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Running, Role: LEADER
I20240502 14:14:59.456364 30448 consensus_queue.cc:237] T fc6401f0ab8c485399325df7c8dd96ad P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:14:59.462256 26283 catalog_manager.cc:5484] T fc6401f0ab8c485399325df7c8dd96ad P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200). New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:14:59.463795 26280 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:14:59.464015 26280 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:14:59.467303 26280 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet b41fb969347644adb57a29ebdb2e6301
I20240502 14:14:59.468461 29289 tablet_service.cc:1514] Processing DeleteTablet for tablet b41fb969347644adb57a29ebdb2e6301 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:59 UTC) from {username='slave'} at 127.0.0.1:59160
I20240502 14:14:59.468783 29421 tablet_service.cc:1514] Processing DeleteTablet for tablet b41fb969347644adb57a29ebdb2e6301 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:59 UTC) from {username='slave'} at 127.0.0.1:40652
I20240502 14:14:59.468827 29553 tablet_service.cc:1514] Processing DeleteTablet for tablet b41fb969347644adb57a29ebdb2e6301 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:59 UTC) from {username='slave'} at 127.0.0.1:36398
I20240502 14:14:59.484941 31123 tablet_replica.cc:330] T b41fb969347644adb57a29ebdb2e6301 P c1100e16b3b9495ca61d4f95e89b6a89: stopping tablet replica
I20240502 14:14:59.485330 31123 raft_consensus.cc:2229] T b41fb969347644adb57a29ebdb2e6301 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:59.486002 31123 raft_consensus.cc:2258] T b41fb969347644adb57a29ebdb2e6301 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:59.489439 31123 ts_tablet_manager.cc:1875] T b41fb969347644adb57a29ebdb2e6301 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:59.504532 31123 ts_tablet_manager.cc:1888] T b41fb969347644adb57a29ebdb2e6301 P c1100e16b3b9495ca61d4f95e89b6a89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:59.505002 31123 log.cc:1196] T b41fb969347644adb57a29ebdb2e6301 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal/wals/b41fb969347644adb57a29ebdb2e6301
I20240502 14:14:59.505930 31123 ts_tablet_manager.cc:1909] T b41fb969347644adb57a29ebdb2e6301 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting consensus metadata
I20240502 14:14:59.507544 26271 catalog_manager.cc:4830] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): tablet b41fb969347644adb57a29ebdb2e6301 (table rebalancer_dynamic_table_new_name [id=36a7544a69604c28ae0b948c3186d44a]) successfully deleted
I20240502 14:14:59.507695 31124 tablet_replica.cc:330] T b41fb969347644adb57a29ebdb2e6301 P 68a3c2b17ed94e06aba66ed4178132ab: stopping tablet replica
I20240502 14:14:59.507899 31125 tablet_replica.cc:330] T b41fb969347644adb57a29ebdb2e6301 P 4092902b3e674857a0147f1b4f2cdb86: stopping tablet replica
I20240502 14:14:59.507941 31124 raft_consensus.cc:2229] T b41fb969347644adb57a29ebdb2e6301 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:59.508352 31125 raft_consensus.cc:2229] T b41fb969347644adb57a29ebdb2e6301 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:59.508687 31125 raft_consensus.cc:2258] T b41fb969347644adb57a29ebdb2e6301 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:59.509744 31125 ts_tablet_manager.cc:1875] T b41fb969347644adb57a29ebdb2e6301 P 4092902b3e674857a0147f1b4f2cdb86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:59.512917 31124 raft_consensus.cc:2258] T b41fb969347644adb57a29ebdb2e6301 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:59.515722 31124 ts_tablet_manager.cc:1875] T b41fb969347644adb57a29ebdb2e6301 P 68a3c2b17ed94e06aba66ed4178132ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:59.516348 31125 ts_tablet_manager.cc:1888] T b41fb969347644adb57a29ebdb2e6301 P 4092902b3e674857a0147f1b4f2cdb86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:59.516573 31125 log.cc:1196] T b41fb969347644adb57a29ebdb2e6301 P 4092902b3e674857a0147f1b4f2cdb86: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/wal/wals/b41fb969347644adb57a29ebdb2e6301
I20240502 14:14:59.517130 31125 ts_tablet_manager.cc:1909] T b41fb969347644adb57a29ebdb2e6301 P 4092902b3e674857a0147f1b4f2cdb86: Deleting consensus metadata
I20240502 14:14:59.518445 26272 catalog_manager.cc:4830] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097): tablet b41fb969347644adb57a29ebdb2e6301 (table rebalancer_dynamic_table_new_name [id=36a7544a69604c28ae0b948c3186d44a]) successfully deleted
I20240502 14:14:59.524547 31124 ts_tablet_manager.cc:1888] T b41fb969347644adb57a29ebdb2e6301 P 68a3c2b17ed94e06aba66ed4178132ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:59.524850 31124 log.cc:1196] T b41fb969347644adb57a29ebdb2e6301 P 68a3c2b17ed94e06aba66ed4178132ab: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal/wals/b41fb969347644adb57a29ebdb2e6301
I20240502 14:14:59.525614 31124 ts_tablet_manager.cc:1909] T b41fb969347644adb57a29ebdb2e6301 P 68a3c2b17ed94e06aba66ed4178132ab: Deleting consensus metadata
I20240502 14:14:59.527143 26272 catalog_manager.cc:4830] TS 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): tablet b41fb969347644adb57a29ebdb2e6301 (table rebalancer_dynamic_table_new_name [id=36a7544a69604c28ae0b948c3186d44a]) successfully deleted
I20240502 14:14:59.569792 26283 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:14:59.584475 29289 tablet_service.cc:1467] Processing CreateTablet for tablet 161a5312d3814a70adbfd85a975610e6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=315e4f643db0467e8e70f8fc76918319]), partition=
I20240502 14:14:59.585008 29289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 161a5312d3814a70adbfd85a975610e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:59.585414 26538 tablet_service.cc:1467] Processing CreateTablet for tablet 161a5312d3814a70adbfd85a975610e6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=315e4f643db0467e8e70f8fc76918319]), partition=
I20240502 14:14:59.585685 29156 tablet_service.cc:1467] Processing CreateTablet for tablet 161a5312d3814a70adbfd85a975610e6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=315e4f643db0467e8e70f8fc76918319]), partition=
I20240502 14:14:59.585896 26538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 161a5312d3814a70adbfd85a975610e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:59.586081 29156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 161a5312d3814a70adbfd85a975610e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:59.594547 31057 tablet_bootstrap.cc:492] T 161a5312d3814a70adbfd85a975610e6 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:14:59.596637 31057 tablet_bootstrap.cc:654] T 161a5312d3814a70adbfd85a975610e6 P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:59.604326 31057 tablet_bootstrap.cc:492] T 161a5312d3814a70adbfd85a975610e6 P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:14:59.604561 31057 ts_tablet_manager.cc:1367] T 161a5312d3814a70adbfd85a975610e6 P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20240502 14:14:59.606113 31057 raft_consensus.cc:348] T 161a5312d3814a70adbfd85a975610e6 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:59.606464 31057 raft_consensus.cc:374] T 161a5312d3814a70adbfd85a975610e6 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:59.606657 31057 raft_consensus.cc:729] T 161a5312d3814a70adbfd85a975610e6 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:14:59.606985 31057 consensus_queue.cc:260] T 161a5312d3814a70adbfd85a975610e6 P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:59.607801 31057 ts_tablet_manager.cc:1398] T 161a5312d3814a70adbfd85a975610e6 P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20240502 14:14:59.614316 31143 tablet_bootstrap.cc:492] T 161a5312d3814a70adbfd85a975610e6 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:14:59.616528 31143 tablet_bootstrap.cc:654] T 161a5312d3814a70adbfd85a975610e6 P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:59.618186 31142 tablet_bootstrap.cc:492] T 161a5312d3814a70adbfd85a975610e6 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:14:59.621568 31142 tablet_bootstrap.cc:654] T 161a5312d3814a70adbfd85a975610e6 P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:59.624855 31143 tablet_bootstrap.cc:492] T 161a5312d3814a70adbfd85a975610e6 P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:14:59.625098 31143 ts_tablet_manager.cc:1367] T 161a5312d3814a70adbfd85a975610e6 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.004s
I20240502 14:14:59.625926 31143 raft_consensus.cc:348] T 161a5312d3814a70adbfd85a975610e6 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:59.626245 31143 raft_consensus.cc:374] T 161a5312d3814a70adbfd85a975610e6 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:59.626457 31143 raft_consensus.cc:729] T 161a5312d3814a70adbfd85a975610e6 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:14:59.626839 31143 consensus_queue.cc:260] T 161a5312d3814a70adbfd85a975610e6 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:59.627878 31143 ts_tablet_manager.cc:1398] T 161a5312d3814a70adbfd85a975610e6 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:14:59.642068 31142 tablet_bootstrap.cc:492] T 161a5312d3814a70adbfd85a975610e6 P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:14:59.642379 31142 ts_tablet_manager.cc:1367] T 161a5312d3814a70adbfd85a975610e6 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.005s
I20240502 14:14:59.643487 31142 raft_consensus.cc:348] T 161a5312d3814a70adbfd85a975610e6 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:59.643841 31142 raft_consensus.cc:374] T 161a5312d3814a70adbfd85a975610e6 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:59.644068 31142 raft_consensus.cc:729] T 161a5312d3814a70adbfd85a975610e6 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:14:59.644371 31142 consensus_queue.cc:260] T 161a5312d3814a70adbfd85a975610e6 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:59.645001 31142 ts_tablet_manager.cc:1398] T 161a5312d3814a70adbfd85a975610e6 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:14:59.678823 30953 raft_consensus.cc:482] T 161a5312d3814a70adbfd85a975610e6 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:59.679054 30953 raft_consensus.cc:504] T 161a5312d3814a70adbfd85a975610e6 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:59.679786 30953 leader_election.cc:288] T 161a5312d3814a70adbfd85a975610e6 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:14:59.680850 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "161a5312d3814a70adbfd85a975610e6" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:14:59.681144 26558 raft_consensus.cc:2454] T 161a5312d3814a70adbfd85a975610e6 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 0.
I20240502 14:14:59.682133 29111 leader_election.cc:302] T 161a5312d3814a70adbfd85a975610e6 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:59.683260 29308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "161a5312d3814a70adbfd85a975610e6" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:14:59.683614 29308 raft_consensus.cc:2454] T 161a5312d3814a70adbfd85a975610e6 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 0.
I20240502 14:14:59.685331 30953 raft_consensus.cc:2781] T 161a5312d3814a70adbfd85a975610e6 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:59.685508 30953 raft_consensus.cc:482] T 161a5312d3814a70adbfd85a975610e6 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:59.685890 30953 raft_consensus.cc:3037] T 161a5312d3814a70adbfd85a975610e6 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:59.695335 30953 raft_consensus.cc:504] T 161a5312d3814a70adbfd85a975610e6 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:59.696082 30953 leader_election.cc:288] T 161a5312d3814a70adbfd85a975610e6 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 election: Requested vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:14:59.696671 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "161a5312d3814a70adbfd85a975610e6" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:14:59.696938 26558 raft_consensus.cc:3037] T 161a5312d3814a70adbfd85a975610e6 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:59.702030 26558 raft_consensus.cc:2454] T 161a5312d3814a70adbfd85a975610e6 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 1.
I20240502 14:14:59.702808 29111 leader_election.cc:302] T 161a5312d3814a70adbfd85a975610e6 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:14:59.706588 29308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "161a5312d3814a70adbfd85a975610e6" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
I20240502 14:14:59.706843 29308 raft_consensus.cc:3037] T 161a5312d3814a70adbfd85a975610e6 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:59.707032 26668 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:14:59.709806 29156 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:14:59.710228 29308 raft_consensus.cc:2454] T 161a5312d3814a70adbfd85a975610e6 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 1.
I20240502 14:14:59.704548 29553 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:14:59.712872 30953 raft_consensus.cc:2781] T 161a5312d3814a70adbfd85a975610e6 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:59.713594 30953 raft_consensus.cc:686] T 161a5312d3814a70adbfd85a975610e6 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 LEADER]: Becoming Leader. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Running, Role: LEADER
I20240502 14:14:59.713961 30953 consensus_queue.cc:237] T 161a5312d3814a70adbfd85a975610e6 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:14:59.717931 26285 catalog_manager.cc:5484] T 161a5312d3814a70adbfd85a975610e6 P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197). New cstate: current_term: 1 leader_uuid: "1cb67158133e4648a6eceba32d4dbba3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:14:59.744702 29421 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:14:59.773321 26538 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:14:59.812091 26406 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:14:59.832527 26285 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:14:59.835949 29289 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:14:59.840047 30953 consensus_queue.cc:1035] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:59.842509 26797 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:14:59.855762 26557 raft_consensus.cc:1261] T 161a5312d3814a70adbfd85a975610e6 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:59.856019 29308 raft_consensus.cc:1261] T 161a5312d3814a70adbfd85a975610e6 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:59.856905 30274 consensus_queue.cc:1035] T 161a5312d3814a70adbfd85a975610e6 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:59.857448 30274 consensus_queue.cc:1035] T 161a5312d3814a70adbfd85a975610e6 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:59.871410 30791 consensus_queue.cc:1035] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:59.878163 30953 consensus_queue.cc:1035] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:59.952086 26280 catalog_manager.cc:5576] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025) does not have the latest schema for tablet 161a5312d3814a70adbfd85a975610e6 (table rebalancer_dynamic_table [id=315e4f643db0467e8e70f8fc76918319]). Expected version 1 got 0
I20240502 14:14:59.968289 31182 consensus_queue.cc:1035] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:59.981613 30448 consensus_queue.cc:1035] T fc6401f0ab8c485399325df7c8dd96ad P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:59.997766 30448 consensus_queue.cc:1035] T fc6401f0ab8c485399325df7c8dd96ad P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:00.022356 31184 tablet.cc:1722] T 161a5312d3814a70adbfd85a975610e6 P 1cb67158133e4648a6eceba32d4dbba3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:00.023126 31184 tablet.cc:1620] T 161a5312d3814a70adbfd85a975610e6 P 1cb67158133e4648a6eceba32d4dbba3: MemRowSet was empty: no flush needed.
I20240502 14:15:00.026175 26276 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_extra_table_22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:00.038480 29289 tablet_service.cc:1467] Processing CreateTablet for tablet d344462241604470a48a443b396177a7 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=10bc76506bf848969a3acfd452e07b7f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:00.039016 29289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d344462241604470a48a443b396177a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:00.040625 26285 catalog_manager.cc:5576] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659) does not have the latest schema for tablet 161a5312d3814a70adbfd85a975610e6 (table rebalancer_dynamic_table [id=315e4f643db0467e8e70f8fc76918319]). Expected version 1 got 0
I20240502 14:15:00.043185 26406 tablet_service.cc:1467] Processing CreateTablet for tablet d344462241604470a48a443b396177a7 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=10bc76506bf848969a3acfd452e07b7f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:00.043768 26406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d344462241604470a48a443b396177a7. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 14:15:00.045965 26754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.198:43659, user_credentials={real_user=slave}} blocked reactor thread for 75067.1us
I20240502 14:15:00.049919 26668 tablet_service.cc:1467] Processing CreateTablet for tablet 924ce5625bf34a739169b005ee4bedc2 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=10bc76506bf848969a3acfd452e07b7f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:00.050513 26668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 924ce5625bf34a739169b005ee4bedc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:00.050568 29156 tablet_service.cc:1467] Processing CreateTablet for tablet d344462241604470a48a443b396177a7 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=10bc76506bf848969a3acfd452e07b7f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:00.051007 29156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d344462241604470a48a443b396177a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:00.057857 29287 tablet_service.cc:1467] Processing CreateTablet for tablet 924ce5625bf34a739169b005ee4bedc2 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=10bc76506bf848969a3acfd452e07b7f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:00.058418 29287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 924ce5625bf34a739169b005ee4bedc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:00.058663 31057 tablet_bootstrap.cc:492] T d344462241604470a48a443b396177a7 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:15:00.061132 31057 tablet_bootstrap.cc:654] T d344462241604470a48a443b396177a7 P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:00.062922 26407 tablet_service.cc:1467] Processing CreateTablet for tablet 924ce5625bf34a739169b005ee4bedc2 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=10bc76506bf848969a3acfd452e07b7f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:00.063297 29421 tablet_service.cc:1467] Processing CreateTablet for tablet 9671eeb3b0e44730a4b0c2849e6ee012 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=10bc76506bf848969a3acfd452e07b7f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:00.063690 26407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 924ce5625bf34a739169b005ee4bedc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:00.063781 29421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9671eeb3b0e44730a4b0c2849e6ee012. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:00.071831 31057 tablet_bootstrap.cc:492] T d344462241604470a48a443b396177a7 P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:15:00.072154 31057 ts_tablet_manager.cc:1367] T d344462241604470a48a443b396177a7 P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.009s
I20240502 14:15:00.073261 31057 raft_consensus.cc:348] T d344462241604470a48a443b396177a7 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:00.073670 31057 raft_consensus.cc:374] T d344462241604470a48a443b396177a7 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:00.073814 31057 raft_consensus.cc:729] T d344462241604470a48a443b396177a7 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:15:00.074318 31057 consensus_queue.cc:260] T d344462241604470a48a443b396177a7 P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:00.075450 31057 ts_tablet_manager.cc:1398] T d344462241604470a48a443b396177a7 P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:15:00.076983 31193 tablet_bootstrap.cc:492] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:15:00.078516 31195 tablet_bootstrap.cc:492] T 924ce5625bf34a739169b005ee4bedc2 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:15:00.079448 31193 tablet_bootstrap.cc:654] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:00.094722 29553 tablet_service.cc:1467] Processing CreateTablet for tablet 9671eeb3b0e44730a4b0c2849e6ee012 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=10bc76506bf848969a3acfd452e07b7f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:00.095227 29553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9671eeb3b0e44730a4b0c2849e6ee012. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:00.111507 31195 tablet_bootstrap.cc:654] T 924ce5625bf34a739169b005ee4bedc2 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:00.113219 31201 tablet_bootstrap.cc:492] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:15:00.114019 31196 tablet_bootstrap.cc:492] T 9671eeb3b0e44730a4b0c2849e6ee012 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:15:00.122571 31201 tablet_bootstrap.cc:654] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:00.122937 29288 tablet_service.cc:1467] Processing CreateTablet for tablet 9671eeb3b0e44730a4b0c2849e6ee012 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=10bc76506bf848969a3acfd452e07b7f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:00.123456 29288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9671eeb3b0e44730a4b0c2849e6ee012. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:00.123515 31142 tablet_bootstrap.cc:492] T 924ce5625bf34a739169b005ee4bedc2 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:15:00.126219 31142 tablet_bootstrap.cc:654] T 924ce5625bf34a739169b005ee4bedc2 P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:00.151278 31197 tablet.cc:1722] T 161a5312d3814a70adbfd85a975610e6 P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:00.152241 31197 tablet.cc:1620] T 161a5312d3814a70adbfd85a975610e6 P c1100e16b3b9495ca61d4f95e89b6a89: MemRowSet was empty: no flush needed.
I20240502 14:15:00.158694 31142 tablet_bootstrap.cc:492] T 924ce5625bf34a739169b005ee4bedc2 P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:15:00.159011 31142 ts_tablet_manager.cc:1367] T 924ce5625bf34a739169b005ee4bedc2 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.006s
I20240502 14:15:00.160121 31142 raft_consensus.cc:348] T 924ce5625bf34a739169b005ee4bedc2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:00.160792 31142 raft_consensus.cc:374] T 924ce5625bf34a739169b005ee4bedc2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:00.160965 31142 raft_consensus.cc:729] T 924ce5625bf34a739169b005ee4bedc2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:15:00.161352 31142 consensus_queue.cc:260] T 924ce5625bf34a739169b005ee4bedc2 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:00.162194 31142 ts_tablet_manager.cc:1398] T 924ce5625bf34a739169b005ee4bedc2 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:15:00.162662 31142 tablet_bootstrap.cc:492] T d344462241604470a48a443b396177a7 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:15:00.172858 26285 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=315e4f643db0467e8e70f8fc76918319] alter complete (version 1)
I20240502 14:15:00.177186 31199 tablet.cc:1722] T 161a5312d3814a70adbfd85a975610e6 P ba187e38a2754d60a12de9ac4856430a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:00.178136 31196 tablet_bootstrap.cc:654] T 9671eeb3b0e44730a4b0c2849e6ee012 P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:00.178158 31199 tablet.cc:1620] T 161a5312d3814a70adbfd85a975610e6 P ba187e38a2754d60a12de9ac4856430a: MemRowSet was empty: no flush needed.
W20240502 14:15:00.181774 26625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.196:32843, user_credentials={real_user=slave}} blocked reactor thread for 61069.8us
W20240502 14:15:00.183063 26624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.194:36025, user_credentials={real_user=slave}} blocked reactor thread for 97864.9us
I20240502 14:15:00.186962 31195 tablet_bootstrap.cc:492] T 924ce5625bf34a739169b005ee4bedc2 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:15:00.187214 31195 ts_tablet_manager.cc:1367] T 924ce5625bf34a739169b005ee4bedc2 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.109s user 0.000s sys 0.004s
I20240502 14:15:00.188318 31195 raft_consensus.cc:348] T 924ce5625bf34a739169b005ee4bedc2 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:00.188656 31195 raft_consensus.cc:374] T 924ce5625bf34a739169b005ee4bedc2 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:00.188803 31195 raft_consensus.cc:729] T 924ce5625bf34a739169b005ee4bedc2 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:15:00.189141 31195 consensus_queue.cc:260] T 924ce5625bf34a739169b005ee4bedc2 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:00.189532 31142 tablet_bootstrap.cc:654] T d344462241604470a48a443b396177a7 P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:00.189824 31195 ts_tablet_manager.cc:1398] T 924ce5625bf34a739169b005ee4bedc2 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:00.192185 26285 catalog_manager.cc:5576] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659) does not have the latest schema for tablet 161a5312d3814a70adbfd85a975610e6 (table rebalancer_dynamic_table [id=315e4f643db0467e8e70f8fc76918319]). Expected version 1 got 0
W20240502 14:15:00.195755 29510 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.194:36025, user_credentials={real_user=slave}} blocked reactor thread for 83047.2us
I20240502 14:15:00.198930 31193 tablet_bootstrap.cc:492] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:15:00.199163 31193 ts_tablet_manager.cc:1367] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.122s user 0.029s sys 0.038s
I20240502 14:15:00.199599 31196 tablet_bootstrap.cc:492] T 9671eeb3b0e44730a4b0c2849e6ee012 P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:15:00.199818 31196 ts_tablet_manager.cc:1367] T 9671eeb3b0e44730a4b0c2849e6ee012 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.086s user 0.005s sys 0.000s
I20240502 14:15:00.200071 31193 raft_consensus.cc:348] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:00.200364 31193 raft_consensus.cc:374] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:00.200539 31193 raft_consensus.cc:729] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:15:00.200920 31193 consensus_queue.cc:260] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:00.201731 31193 ts_tablet_manager.cc:1398] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:00.206017 31196 raft_consensus.cc:348] T 9671eeb3b0e44730a4b0c2849e6ee012 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:00.206146 31195 tablet_bootstrap.cc:492] T d344462241604470a48a443b396177a7 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:15:00.206307 31196 raft_consensus.cc:374] T 9671eeb3b0e44730a4b0c2849e6ee012 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:00.206471 31196 raft_consensus.cc:729] T 9671eeb3b0e44730a4b0c2849e6ee012 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:15:00.206805 31196 consensus_queue.cc:260] T 9671eeb3b0e44730a4b0c2849e6ee012 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:00.207397 31196 ts_tablet_manager.cc:1398] T 9671eeb3b0e44730a4b0c2849e6ee012 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.007s user 0.000s sys 0.000s
I20240502 14:15:00.207943 30947 raft_consensus.cc:482] T 924ce5625bf34a739169b005ee4bedc2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:00.208103 30947 raft_consensus.cc:504] T 924ce5625bf34a739169b005ee4bedc2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:00.209197 30947 leader_election.cc:288] T 924ce5625bf34a739169b005ee4bedc2 P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:15:00.210443 31195 tablet_bootstrap.cc:654] T d344462241604470a48a443b396177a7 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:00.210695 26427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "924ce5625bf34a739169b005ee4bedc2" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:15:00.210968 26427 raft_consensus.cc:2454] T 924ce5625bf34a739169b005ee4bedc2 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 0.
I20240502 14:15:00.214774 31201 tablet_bootstrap.cc:492] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:15:00.214964 31201 ts_tablet_manager.cc:1367] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.102s user 0.002s sys 0.001s
I20240502 14:15:00.215411 29244 leader_election.cc:302] T 924ce5625bf34a739169b005ee4bedc2 P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:15:00.215799 31201 raft_consensus.cc:348] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:00.216035 31201 raft_consensus.cc:374] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:00.216150 31201 raft_consensus.cc:729] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:15:00.216408 31201 consensus_queue.cc:260] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:00.216789 30947 raft_consensus.cc:2781] T 924ce5625bf34a739169b005ee4bedc2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:00.216917 30947 raft_consensus.cc:482] T 924ce5625bf34a739169b005ee4bedc2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:00.217000 30947 raft_consensus.cc:3037] T 924ce5625bf34a739169b005ee4bedc2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:00.217032 31201 ts_tablet_manager.cc:1398] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:15:00.219947 30947 raft_consensus.cc:504] T 924ce5625bf34a739169b005ee4bedc2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:00.225203 26427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "924ce5625bf34a739169b005ee4bedc2" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:15:00.225472 26427 raft_consensus.cc:3037] T 924ce5625bf34a739169b005ee4bedc2 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:00.230193 31195 tablet_bootstrap.cc:492] T d344462241604470a48a443b396177a7 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:15:00.230413 31195 ts_tablet_manager.cc:1367] T d344462241604470a48a443b396177a7 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.024s user 0.010s sys 0.000s
I20240502 14:15:00.231271 31195 raft_consensus.cc:348] T d344462241604470a48a443b396177a7 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:00.231514 31195 raft_consensus.cc:374] T d344462241604470a48a443b396177a7 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:00.231618 31195 raft_consensus.cc:729] T d344462241604470a48a443b396177a7 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:15:00.231932 31195 consensus_queue.cc:260] T d344462241604470a48a443b396177a7 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:00.232765 31195 ts_tablet_manager.cc:1398] T d344462241604470a48a443b396177a7 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:00.233068 31142 tablet_bootstrap.cc:492] T d344462241604470a48a443b396177a7 P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:15:00.233243 31142 ts_tablet_manager.cc:1367] T d344462241604470a48a443b396177a7 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.071s user 0.002s sys 0.003s
I20240502 14:15:00.233875 31142 raft_consensus.cc:348] T d344462241604470a48a443b396177a7 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:00.234045 31142 raft_consensus.cc:374] T d344462241604470a48a443b396177a7 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:00.234146 31142 raft_consensus.cc:729] T d344462241604470a48a443b396177a7 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:15:00.235927 31142 consensus_queue.cc:260] T d344462241604470a48a443b396177a7 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:00.237277 26427 raft_consensus.cc:2454] T 924ce5625bf34a739169b005ee4bedc2 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 1.
I20240502 14:15:00.238575 31142 ts_tablet_manager.cc:1398] T d344462241604470a48a443b396177a7 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 14:15:00.241057 31205 raft_consensus.cc:482] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:00.241242 31205 raft_consensus.cc:504] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:00.242066 31205 leader_election.cc:288] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:15:00.239071 30947 leader_election.cc:288] T 924ce5625bf34a739169b005ee4bedc2 P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 election: Requested vote from peers 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:15:00.243077 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "924ce5625bf34a739169b005ee4bedc2" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:15:00.243372 29309 raft_consensus.cc:2379] T 924ce5625bf34a739169b005ee4bedc2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in current term 1: Already voted for candidate c1100e16b3b9495ca61d4f95e89b6a89 in this term.
I20240502 14:15:00.244364 31142 tablet_bootstrap.cc:492] T 9671eeb3b0e44730a4b0c2849e6ee012 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:15:00.245489 29244 leader_election.cc:302] T 924ce5625bf34a739169b005ee4bedc2 P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:15:00.246505 26423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "924ce5625bf34a739169b005ee4bedc2" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:15:00.246794 26423 raft_consensus.cc:2379] T 924ce5625bf34a739169b005ee4bedc2 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in current term 1: Already voted for candidate c1100e16b3b9495ca61d4f95e89b6a89 in this term.
I20240502 14:15:00.247196 31142 tablet_bootstrap.cc:654] T 9671eeb3b0e44730a4b0c2849e6ee012 P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:00.252241 26624 leader_election.cc:302] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters: 89af7f423cf44717a08d6b142f0ed49d, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:15:00.252704 31205 raft_consensus.cc:3037] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:00.255481 31205 raft_consensus.cc:2726] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:15:00.251114 31217 raft_consensus.cc:2781] T 924ce5625bf34a739169b005ee4bedc2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:00.260072 31217 raft_consensus.cc:686] T 924ce5625bf34a739169b005ee4bedc2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 LEADER]: Becoming Leader. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Running, Role: LEADER
I20240502 14:15:00.260676 31217 consensus_queue.cc:237] T 924ce5625bf34a739169b005ee4bedc2 P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:00.273377 26689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "924ce5625bf34a739169b005ee4bedc2" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:15:00.273655 26689 raft_consensus.cc:2454] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 1.
I20240502 14:15:00.274152 26688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "924ce5625bf34a739169b005ee4bedc2" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:15:00.278970 26688 raft_consensus.cc:2454] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 1.
I20240502 14:15:00.284034 26276 catalog_manager.cc:5576] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025) does not have the latest schema for tablet 161a5312d3814a70adbfd85a975610e6 (table rebalancer_dynamic_table [id=315e4f643db0467e8e70f8fc76918319]). Expected version 1 got 0
I20240502 14:15:00.304940 31142 tablet_bootstrap.cc:492] T 9671eeb3b0e44730a4b0c2849e6ee012 P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:15:00.305351 31142 ts_tablet_manager.cc:1367] T 9671eeb3b0e44730a4b0c2849e6ee012 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.061s user 0.002s sys 0.003s
I20240502 14:15:00.305289 26277 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:15:00.306558 31142 raft_consensus.cc:348] T 9671eeb3b0e44730a4b0c2849e6ee012 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:00.306907 31142 raft_consensus.cc:374] T 9671eeb3b0e44730a4b0c2849e6ee012 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:00.307005 31142 raft_consensus.cc:729] T 9671eeb3b0e44730a4b0c2849e6ee012 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:15:00.307282 31142 consensus_queue.cc:260] T 9671eeb3b0e44730a4b0c2849e6ee012 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:00.307930 31142 ts_tablet_manager.cc:1398] T 9671eeb3b0e44730a4b0c2849e6ee012 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:15:00.318059 26283 catalog_manager.cc:5484] T 924ce5625bf34a739169b005ee4bedc2 P c1100e16b3b9495ca61d4f95e89b6a89 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198). New cstate: current_term: 1 leader_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:00.329980 31184 tablet.cc:1722] T 161a5312d3814a70adbfd85a975610e6 P 1cb67158133e4648a6eceba32d4dbba3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:00.330557 31184 tablet.cc:1620] T 161a5312d3814a70adbfd85a975610e6 P 1cb67158133e4648a6eceba32d4dbba3: MemRowSet was empty: no flush needed.
I20240502 14:15:00.332039 31197 tablet.cc:1722] T 161a5312d3814a70adbfd85a975610e6 P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:00.332590 31197 tablet.cc:1620] T 161a5312d3814a70adbfd85a975610e6 P c1100e16b3b9495ca61d4f95e89b6a89: MemRowSet was empty: no flush needed.
I20240502 14:15:00.342068 31199 tablet.cc:1722] T 161a5312d3814a70adbfd85a975610e6 P ba187e38a2754d60a12de9ac4856430a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:00.342633 31199 tablet.cc:1620] T 161a5312d3814a70adbfd85a975610e6 P ba187e38a2754d60a12de9ac4856430a: MemRowSet was empty: no flush needed.
I20240502 14:15:00.355180 26283 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=315e4f643db0467e8e70f8fc76918319] alter complete (version 2)
I20240502 14:15:00.379285 31173 raft_consensus.cc:482] T d344462241604470a48a443b396177a7 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:00.379489 31173 raft_consensus.cc:504] T d344462241604470a48a443b396177a7 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:00.380270 31173 leader_election.cc:288] T d344462241604470a48a443b396177a7 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:15:00.380712 29174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d344462241604470a48a443b396177a7" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:15:00.380957 29174 raft_consensus.cc:2454] T d344462241604470a48a443b396177a7 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:15:00.381017 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d344462241604470a48a443b396177a7" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:15:00.381247 29309 raft_consensus.cc:2454] T d344462241604470a48a443b396177a7 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:15:00.381546 26362 leader_election.cc:302] T d344462241604470a48a443b396177a7 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:15:00.381834 31173 raft_consensus.cc:2781] T d344462241604470a48a443b396177a7 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:00.381968 31173 raft_consensus.cc:482] T d344462241604470a48a443b396177a7 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:00.382073 31173 raft_consensus.cc:3037] T d344462241604470a48a443b396177a7 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:00.385252 31173 raft_consensus.cc:504] T d344462241604470a48a443b396177a7 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:00.386530 29174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d344462241604470a48a443b396177a7" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:15:00.386682 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d344462241604470a48a443b396177a7" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
I20240502 14:15:00.386765 29174 raft_consensus.cc:3037] T d344462241604470a48a443b396177a7 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:00.386885 29309 raft_consensus.cc:3037] T d344462241604470a48a443b396177a7 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:00.389662 29309 raft_consensus.cc:2454] T d344462241604470a48a443b396177a7 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:15:00.386034 31173 leader_election.cc:288] T d344462241604470a48a443b396177a7 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Requested vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:15:00.390588 29174 raft_consensus.cc:2454] T d344462241604470a48a443b396177a7 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:15:00.391590 26362 leader_election.cc:302] T d344462241604470a48a443b396177a7 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:15:00.391945 31173 raft_consensus.cc:2781] T d344462241604470a48a443b396177a7 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:00.392527 31173 raft_consensus.cc:686] T d344462241604470a48a443b396177a7 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Becoming Leader. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Running, Role: LEADER
I20240502 14:15:00.392908 31173 consensus_queue.cc:237] T d344462241604470a48a443b396177a7 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:00.402810 26283 catalog_manager.cc:5484] T d344462241604470a48a443b396177a7 P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: term changed from 0 to 1, leader changed from <none> to 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193). New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:15:00.435642 26819 consensus_queue.cc:237] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:00.439646 29309 raft_consensus.cc:1261] T 6b4745b1153b4acd999ac3a5c137ff53 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:00.440050 26427 raft_consensus.cc:1261] T 6b4745b1153b4acd999ac3a5c137ff53 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:00.440631 31191 consensus_queue.cc:1035] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:00.441215 31191 consensus_queue.cc:1035] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:00.444664 31191 raft_consensus.cc:2932] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:15:00.449309 26427 raft_consensus.cc:2932] T 6b4745b1153b4acd999ac3a5c137ff53 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:15:00.449599 29307 raft_consensus.cc:2932] T 6b4745b1153b4acd999ac3a5c137ff53 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:15:00.456349 26283 catalog_manager.cc:5484] T 6b4745b1153b4acd999ac3a5c137ff53 P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: config changed from index 4 to 5, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) evicted. New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:15:00.458868 26277 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:15:00.460573 26538 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b4745b1153b4acd999ac3a5c137ff53 with delete_type TABLET_DATA_TOMBSTONED (TS ba187e38a2754d60a12de9ac4856430a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51696
I20240502 14:15:00.462015 31241 tablet_replica.cc:330] T 6b4745b1153b4acd999ac3a5c137ff53 P ba187e38a2754d60a12de9ac4856430a: stopping tablet replica
I20240502 14:15:00.462352 31241 raft_consensus.cc:2229] T 6b4745b1153b4acd999ac3a5c137ff53 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:00.462759 31241 raft_consensus.cc:2258] T 6b4745b1153b4acd999ac3a5c137ff53 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:00.463740 31241 ts_tablet_manager.cc:1875] T 6b4745b1153b4acd999ac3a5c137ff53 P ba187e38a2754d60a12de9ac4856430a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:00.471252 31241 ts_tablet_manager.cc:1888] T 6b4745b1153b4acd999ac3a5c137ff53 P ba187e38a2754d60a12de9ac4856430a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:15:00.471544 31241 log.cc:1196] T 6b4745b1153b4acd999ac3a5c137ff53 P ba187e38a2754d60a12de9ac4856430a: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/6b4745b1153b4acd999ac3a5c137ff53
I20240502 14:15:00.472724 26271 catalog_manager.cc:4830] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): tablet 6b4745b1153b4acd999ac3a5c137ff53 (table rebalance_test_table_15 [id=ff3107a4d4c94012a52f0cd0ca9ef235]) successfully deleted
I20240502 14:15:00.476013 31184 tablet.cc:1722] T 161a5312d3814a70adbfd85a975610e6 P 1cb67158133e4648a6eceba32d4dbba3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:00.476907 31199 tablet.cc:1722] T 161a5312d3814a70adbfd85a975610e6 P ba187e38a2754d60a12de9ac4856430a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:00.482722 31197 tablet.cc:1722] T 161a5312d3814a70adbfd85a975610e6 P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:00.488420 26280 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table_new_name [id=315e4f643db0467e8e70f8fc76918319] alter complete (version 3)
I20240502 14:15:00.528533 31203 raft_consensus.cc:482] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:00.528743 31203 raft_consensus.cc:504] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:00.529548 31203 leader_election.cc:288] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:15:00.530055 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9671eeb3b0e44730a4b0c2849e6ee012" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:15:00.530141 29307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9671eeb3b0e44730a4b0c2849e6ee012" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:15:00.530309 29443 raft_consensus.cc:2454] T 9671eeb3b0e44730a4b0c2849e6ee012 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:15:00.530388 29307 raft_consensus.cc:2454] T 9671eeb3b0e44730a4b0c2849e6ee012 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:15:00.530838 29511 leader_election.cc:302] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, 68a3c2b17ed94e06aba66ed4178132ab; no voters:
I20240502 14:15:00.531167 31203 raft_consensus.cc:2781] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:00.531306 31203 raft_consensus.cc:482] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:00.531476 31203 raft_consensus.cc:3037] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:00.535128 31203 raft_consensus.cc:504] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:00.535897 31203 leader_election.cc:288] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Requested vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:15:00.536329 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9671eeb3b0e44730a4b0c2849e6ee012" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
I20240502 14:15:00.536587 29443 raft_consensus.cc:3037] T 9671eeb3b0e44730a4b0c2849e6ee012 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:00.536505 29307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9671eeb3b0e44730a4b0c2849e6ee012" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
I20240502 14:15:00.536768 29307 raft_consensus.cc:3037] T 9671eeb3b0e44730a4b0c2849e6ee012 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:00.539731 29307 raft_consensus.cc:2454] T 9671eeb3b0e44730a4b0c2849e6ee012 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:15:00.539695 29443 raft_consensus.cc:2454] T 9671eeb3b0e44730a4b0c2849e6ee012 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:15:00.541503 29511 leader_election.cc:302] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, 68a3c2b17ed94e06aba66ed4178132ab; no voters:
I20240502 14:15:00.542244 31203 raft_consensus.cc:2781] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:00.543046 31203 raft_consensus.cc:686] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Becoming Leader. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Running, Role: LEADER
I20240502 14:15:00.543773 31203 consensus_queue.cc:237] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:00.557467 26283 catalog_manager.cc:5484] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200). New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:15:00.591452 26277 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:00.591823 26277 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:00.605273 26277 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet 161a5312d3814a70adbfd85a975610e6
I20240502 14:15:00.606606 29289 tablet_service.cc:1514] Processing DeleteTablet for tablet 161a5312d3814a70adbfd85a975610e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:00 UTC) from {username='slave'} at 127.0.0.1:59160
I20240502 14:15:00.606685 26538 tablet_service.cc:1514] Processing DeleteTablet for tablet 161a5312d3814a70adbfd85a975610e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:00 UTC) from {username='slave'} at 127.0.0.1:51696
I20240502 14:15:00.607127 31241 tablet_replica.cc:330] T 161a5312d3814a70adbfd85a975610e6 P ba187e38a2754d60a12de9ac4856430a: stopping tablet replica
I20240502 14:15:00.607426 31241 raft_consensus.cc:2229] T 161a5312d3814a70adbfd85a975610e6 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:00.607661 31273 tablet_replica.cc:330] T 161a5312d3814a70adbfd85a975610e6 P c1100e16b3b9495ca61d4f95e89b6a89: stopping tablet replica
I20240502 14:15:00.607726 31241 raft_consensus.cc:2258] T 161a5312d3814a70adbfd85a975610e6 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:00.607852 31273 raft_consensus.cc:2229] T 161a5312d3814a70adbfd85a975610e6 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:00.608089 31273 raft_consensus.cc:2258] T 161a5312d3814a70adbfd85a975610e6 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:00.609660 31241 ts_tablet_manager.cc:1875] T 161a5312d3814a70adbfd85a975610e6 P ba187e38a2754d60a12de9ac4856430a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:00.609660 31273 ts_tablet_manager.cc:1875] T 161a5312d3814a70adbfd85a975610e6 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:00.614322 29156 tablet_service.cc:1514] Processing DeleteTablet for tablet 161a5312d3814a70adbfd85a975610e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:00 UTC) from {username='slave'} at 127.0.0.1:55202
I20240502 14:15:00.622977 31273 ts_tablet_manager.cc:1888] T 161a5312d3814a70adbfd85a975610e6 P c1100e16b3b9495ca61d4f95e89b6a89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:00.623206 31241 ts_tablet_manager.cc:1888] T 161a5312d3814a70adbfd85a975610e6 P ba187e38a2754d60a12de9ac4856430a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:00.623296 31273 log.cc:1196] T 161a5312d3814a70adbfd85a975610e6 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal/wals/161a5312d3814a70adbfd85a975610e6
I20240502 14:15:00.623394 31241 log.cc:1196] T 161a5312d3814a70adbfd85a975610e6 P ba187e38a2754d60a12de9ac4856430a: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/161a5312d3814a70adbfd85a975610e6
I20240502 14:15:00.623884 31241 ts_tablet_manager.cc:1909] T 161a5312d3814a70adbfd85a975610e6 P ba187e38a2754d60a12de9ac4856430a: Deleting consensus metadata
I20240502 14:15:00.624142 31273 ts_tablet_manager.cc:1909] T 161a5312d3814a70adbfd85a975610e6 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting consensus metadata
I20240502 14:15:00.625939 31274 tablet_replica.cc:330] T 161a5312d3814a70adbfd85a975610e6 P 1cb67158133e4648a6eceba32d4dbba3: stopping tablet replica
I20240502 14:15:00.625886 26271 catalog_manager.cc:4830] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): tablet 161a5312d3814a70adbfd85a975610e6 (table rebalancer_dynamic_table_new_name [id=315e4f643db0467e8e70f8fc76918319]) successfully deleted
I20240502 14:15:00.626228 31274 raft_consensus.cc:2229] T 161a5312d3814a70adbfd85a975610e6 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:15:00.626819 31274 raft_consensus.cc:2258] T 161a5312d3814a70adbfd85a975610e6 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:00.627964 31274 ts_tablet_manager.cc:1875] T 161a5312d3814a70adbfd85a975610e6 P 1cb67158133e4648a6eceba32d4dbba3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:00.628361 26271 catalog_manager.cc:4830] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): tablet 161a5312d3814a70adbfd85a975610e6 (table rebalancer_dynamic_table_new_name [id=315e4f643db0467e8e70f8fc76918319]) successfully deleted
I20240502 14:15:00.634886 31274 ts_tablet_manager.cc:1888] T 161a5312d3814a70adbfd85a975610e6 P 1cb67158133e4648a6eceba32d4dbba3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:00.635038 31274 log.cc:1196] T 161a5312d3814a70adbfd85a975610e6 P 1cb67158133e4648a6eceba32d4dbba3: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal/wals/161a5312d3814a70adbfd85a975610e6
I20240502 14:15:00.635553 31274 ts_tablet_manager.cc:1909] T 161a5312d3814a70adbfd85a975610e6 P 1cb67158133e4648a6eceba32d4dbba3: Deleting consensus metadata
I20240502 14:15:00.636988 26271 catalog_manager.cc:4830] TS 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): tablet 161a5312d3814a70adbfd85a975610e6 (table rebalancer_dynamic_table_new_name [id=315e4f643db0467e8e70f8fc76918319]) successfully deleted
I20240502 14:15:00.710870 26285 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:15:00.720810 29156 tablet_service.cc:1467] Processing CreateTablet for tablet 92d6ce3fbd1c4cbb9caec04f7eb593e5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c0ff6e6a8c224160b6c0977820de56f9]), partition=
I20240502 14:15:00.720873 29421 tablet_service.cc:1467] Processing CreateTablet for tablet 92d6ce3fbd1c4cbb9caec04f7eb593e5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c0ff6e6a8c224160b6c0977820de56f9]), partition=
I20240502 14:15:00.720999 29289 tablet_service.cc:1467] Processing CreateTablet for tablet 92d6ce3fbd1c4cbb9caec04f7eb593e5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c0ff6e6a8c224160b6c0977820de56f9]), partition=
I20240502 14:15:00.721261 29421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92d6ce3fbd1c4cbb9caec04f7eb593e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:00.721277 29156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92d6ce3fbd1c4cbb9caec04f7eb593e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:00.721333 29289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92d6ce3fbd1c4cbb9caec04f7eb593e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:00.728380 31142 tablet_bootstrap.cc:492] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:15:00.729604 31217 consensus_queue.cc:1035] T 924ce5625bf34a739169b005ee4bedc2 P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:00.729930 31297 tablet_bootstrap.cc:492] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:15:00.730150 31142 tablet_bootstrap.cc:654] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:00.731523 31297 tablet_bootstrap.cc:654] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:00.740115 31297 tablet_bootstrap.cc:492] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:15:00.740324 31297 ts_tablet_manager.cc:1367] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240502 14:15:00.741561 31297 raft_consensus.cc:348] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:00.742025 31297 raft_consensus.cc:374] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:00.742185 31297 raft_consensus.cc:729] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:15:00.742568 31297 consensus_queue.cc:260] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:00.743983 31297 ts_tablet_manager.cc:1398] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240502 14:15:00.746381 31142 tablet_bootstrap.cc:492] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:15:00.746572 31142 ts_tablet_manager.cc:1367] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.005s
I20240502 14:15:00.746703 31302 tablet_bootstrap.cc:492] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:15:00.747972 31142 raft_consensus.cc:348] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:00.748651 31302 tablet_bootstrap.cc:654] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:00.748335 31142 raft_consensus.cc:374] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:00.748996 31142 raft_consensus.cc:729] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:15:00.749411 31142 consensus_queue.cc:260] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:00.749984 30947 consensus_queue.cc:1035] T 924ce5625bf34a739169b005ee4bedc2 P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:00.750401 31142 ts_tablet_manager.cc:1398] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20240502 14:15:00.763737 31302 tablet_bootstrap.cc:492] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:15:00.763993 31302 ts_tablet_manager.cc:1367] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.007s
I20240502 14:15:00.764935 31302 raft_consensus.cc:348] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:00.765210 31302 raft_consensus.cc:374] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:00.765319 31302 raft_consensus.cc:729] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:15:00.765631 31302 consensus_queue.cc:260] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:00.768069 31302 ts_tablet_manager.cc:1398] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:15:00.829579 31181 raft_consensus.cc:482] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:00.829797 31181 raft_consensus.cc:504] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:00.830699 31181 leader_election.cc:288] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:15:00.831221 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d6ce3fbd1c4cbb9caec04f7eb593e5" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:15:00.831509 29443 raft_consensus.cc:2454] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 0.
I20240502 14:15:00.832145 29112 leader_election.cc:302] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 68a3c2b17ed94e06aba66ed4178132ab; no voters:
I20240502 14:15:00.834506 31181 raft_consensus.cc:2781] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:00.834657 31181 raft_consensus.cc:482] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:00.834780 31181 raft_consensus.cc:3037] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:00.839185 29307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d6ce3fbd1c4cbb9caec04f7eb593e5" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:15:00.839277 31181 raft_consensus.cc:504] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:00.839478 29307 raft_consensus.cc:2454] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 0.
I20240502 14:15:00.840929 31181 leader_election.cc:288] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 election: Requested vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:15:00.841053 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d6ce3fbd1c4cbb9caec04f7eb593e5" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
I20240502 14:15:00.841326 29443 raft_consensus.cc:3037] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:00.841634 29307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d6ce3fbd1c4cbb9caec04f7eb593e5" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
I20240502 14:15:00.841853 29307 raft_consensus.cc:3037] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:00.846958 29443 raft_consensus.cc:2454] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 1.
I20240502 14:15:00.846982 29307 raft_consensus.cc:2454] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 1.
I20240502 14:15:00.847635 29111 leader_election.cc:302] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:15:00.848177 31181 raft_consensus.cc:2781] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:00.848371 31181 raft_consensus.cc:686] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 LEADER]: Becoming Leader. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Running, Role: LEADER
I20240502 14:15:00.848753 31181 consensus_queue.cc:237] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:00.851073 31179 consensus_queue.cc:1035] T d344462241604470a48a443b396177a7 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:00.855513 26278 catalog_manager.cc:5484] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197). New cstate: current_term: 1 leader_uuid: "1cb67158133e4648a6eceba32d4dbba3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:15:00.857723 31173 consensus_queue.cc:1035] T d344462241604470a48a443b396177a7 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:00.933615 29156 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:00.936719 26538 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:00.955581 29289 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:15:00.961092 26668 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:15:00.969605 29421 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:15:00.970172 26283 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:15:00.972762 26407 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:00.973323 26797 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:00.982283 29443 raft_consensus.cc:1261] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:00.985721 30953 consensus_queue.cc:1035] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:00.985951 29307 raft_consensus.cc:1261] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:00.989239 30953 consensus_queue.cc:1035] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:01.005707 30448 consensus_queue.cc:1035] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:01.009343 31346 consensus_queue.cc:1035] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 14:15:01.029176 29553 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:01.038158 26283 catalog_manager.cc:5576] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659) does not have the latest schema for tablet 92d6ce3fbd1c4cbb9caec04f7eb593e5 (table rebalancer_dynamic_table [id=c0ff6e6a8c224160b6c0977820de56f9]). Expected version 1 got 0
I20240502 14:15:01.053516 31353 tablet.cc:1722] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:01.053969 31352 tablet.cc:1722] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 1cb67158133e4648a6eceba32d4dbba3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:01.054738 31352 tablet.cc:1620] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 1cb67158133e4648a6eceba32d4dbba3: MemRowSet was empty: no flush needed.
I20240502 14:15:01.055500 31353 tablet.cc:1620] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P c1100e16b3b9495ca61d4f95e89b6a89: MemRowSet was empty: no flush needed.
I20240502 14:15:01.058619 26283 catalog_manager.cc:5576] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659) does not have the latest schema for tablet 92d6ce3fbd1c4cbb9caec04f7eb593e5 (table rebalancer_dynamic_table [id=c0ff6e6a8c224160b6c0977820de56f9]). Expected version 1 got 0
I20240502 14:15:01.065735 26283 catalog_manager.cc:5576] TS 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781) does not have the latest schema for tablet 92d6ce3fbd1c4cbb9caec04f7eb593e5 (table rebalancer_dynamic_table [id=c0ff6e6a8c224160b6c0977820de56f9]). Expected version 1 got 0
I20240502 14:15:01.102115 26283 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=c0ff6e6a8c224160b6c0977820de56f9] alter complete (version 1)
I20240502 14:15:01.135217 26279 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_extra_table_24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:01.139024 31355 tablet.cc:1722] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 68a3c2b17ed94e06aba66ed4178132ab: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:01.142699 31355 tablet.cc:1620] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 68a3c2b17ed94e06aba66ed4178132ab: MemRowSet was empty: no flush needed.
I20240502 14:15:01.148779 26283 catalog_manager.cc:5576] TS 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781) does not have the latest schema for tablet 92d6ce3fbd1c4cbb9caec04f7eb593e5 (table rebalancer_dynamic_table [id=c0ff6e6a8c224160b6c0977820de56f9]). Expected version 1 got 0
I20240502 14:15:01.151213 29289 tablet_service.cc:1467] Processing CreateTablet for tablet 3f6d55d89e55425183da73dbb3f4175f (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=8ce70558d80e412a89355b4917fc1a4e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:01.151291 29421 tablet_service.cc:1467] Processing CreateTablet for tablet 3f6d55d89e55425183da73dbb3f4175f (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=8ce70558d80e412a89355b4917fc1a4e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:01.151372 26538 tablet_service.cc:1467] Processing CreateTablet for tablet 3f6d55d89e55425183da73dbb3f4175f (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=8ce70558d80e412a89355b4917fc1a4e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:01.151729 29421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f6d55d89e55425183da73dbb3f4175f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:01.151731 29289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f6d55d89e55425183da73dbb3f4175f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:01.151768 26538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f6d55d89e55425183da73dbb3f4175f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:01.157817 29423 tablet_service.cc:1467] Processing CreateTablet for tablet be1c8c53ff6843a9bf07d006170f1f57 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=8ce70558d80e412a89355b4917fc1a4e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:01.158391 29423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be1c8c53ff6843a9bf07d006170f1f57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:01.158954 29553 tablet_service.cc:1467] Processing CreateTablet for tablet be1c8c53ff6843a9bf07d006170f1f57 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=8ce70558d80e412a89355b4917fc1a4e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:01.159484 29553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be1c8c53ff6843a9bf07d006170f1f57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:01.164247 26537 tablet_service.cc:1467] Processing CreateTablet for tablet 9d28bfbd29c94d2ebbedbae53231bf1e (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=8ce70558d80e412a89355b4917fc1a4e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:01.164808 26537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d28bfbd29c94d2ebbedbae53231bf1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:01.167284 31356 tablet_bootstrap.cc:492] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:15:01.170145 29288 tablet_service.cc:1467] Processing CreateTablet for tablet 9d28bfbd29c94d2ebbedbae53231bf1e (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=8ce70558d80e412a89355b4917fc1a4e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:01.170644 29288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d28bfbd29c94d2ebbedbae53231bf1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:01.173673 29156 tablet_service.cc:1467] Processing CreateTablet for tablet 9d28bfbd29c94d2ebbedbae53231bf1e (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=8ce70558d80e412a89355b4917fc1a4e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:01.174154 29156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d28bfbd29c94d2ebbedbae53231bf1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:01.176231 31356 tablet_bootstrap.cc:654] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:01.181476 31302 tablet_bootstrap.cc:492] T 9d28bfbd29c94d2ebbedbae53231bf1e P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:15:01.181536 31297 tablet_bootstrap.cc:492] T 3f6d55d89e55425183da73dbb3f4175f P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:15:01.183271 31297 tablet_bootstrap.cc:654] T 3f6d55d89e55425183da73dbb3f4175f P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:01.183267 31302 tablet_bootstrap.cc:654] T 9d28bfbd29c94d2ebbedbae53231bf1e P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:01.185642 31356 tablet_bootstrap.cc:492] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:15:01.185897 31356 ts_tablet_manager.cc:1367] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.000s
I20240502 14:15:01.186918 31356 raft_consensus.cc:348] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:01.187232 31356 raft_consensus.cc:374] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:01.187367 31356 raft_consensus.cc:729] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:15:01.187683 26797 tablet_service.cc:1467] Processing CreateTablet for tablet be1c8c53ff6843a9bf07d006170f1f57 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=8ce70558d80e412a89355b4917fc1a4e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:01.188220 26797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be1c8c53ff6843a9bf07d006170f1f57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:01.187693 31356 consensus_queue.cc:260] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:01.191354 31356 ts_tablet_manager.cc:1398] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:15:01.200130 31361 tablet_bootstrap.cc:492] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:15:01.206701 31361 tablet_bootstrap.cc:654] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:01.208276 26283 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:15:01.212455 31361 tablet_bootstrap.cc:492] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:15:01.212728 31361 ts_tablet_manager.cc:1367] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.001s
I20240502 14:15:01.213656 31361 raft_consensus.cc:348] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:01.213968 31361 raft_consensus.cc:374] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:01.214192 31361 raft_consensus.cc:729] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:15:01.214478 31361 consensus_queue.cc:260] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:01.215469 31361 ts_tablet_manager.cc:1398] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:15:01.223006 31352 tablet.cc:1722] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 1cb67158133e4648a6eceba32d4dbba3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:01.223323 31353 tablet.cc:1722] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:01.223568 31352 tablet.cc:1620] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 1cb67158133e4648a6eceba32d4dbba3: MemRowSet was empty: no flush needed.
I20240502 14:15:01.223873 31353 tablet.cc:1620] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P c1100e16b3b9495ca61d4f95e89b6a89: MemRowSet was empty: no flush needed.
I20240502 14:15:01.270454 26276 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=c0ff6e6a8c224160b6c0977820de56f9] alter complete (version 2)
I20240502 14:15:01.271584 31364 tablet_bootstrap.cc:492] T 9d28bfbd29c94d2ebbedbae53231bf1e P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:15:01.273535 31364 tablet_bootstrap.cc:654] T 9d28bfbd29c94d2ebbedbae53231bf1e P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:01.284075 31364 tablet_bootstrap.cc:492] T 9d28bfbd29c94d2ebbedbae53231bf1e P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:15:01.286202 31364 ts_tablet_manager.cc:1367] T 9d28bfbd29c94d2ebbedbae53231bf1e P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.005s
I20240502 14:15:01.287173 31364 raft_consensus.cc:348] T 9d28bfbd29c94d2ebbedbae53231bf1e P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:01.287524 31364 raft_consensus.cc:374] T 9d28bfbd29c94d2ebbedbae53231bf1e P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:01.287680 31364 raft_consensus.cc:729] T 9d28bfbd29c94d2ebbedbae53231bf1e P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:15:01.288012 31364 consensus_queue.cc:260] T 9d28bfbd29c94d2ebbedbae53231bf1e P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:01.289376 31356 tablet_bootstrap.cc:492] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:15:01.290232 31364 ts_tablet_manager.cc:1398] T 9d28bfbd29c94d2ebbedbae53231bf1e P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 14:15:01.290508 31364 tablet_bootstrap.cc:492] T 3f6d55d89e55425183da73dbb3f4175f P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:15:01.292125 31356 tablet_bootstrap.cc:654] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:01.292289 31364 tablet_bootstrap.cc:654] T 3f6d55d89e55425183da73dbb3f4175f P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:01.294672 31356 tablet_bootstrap.cc:492] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:15:01.294852 31356 ts_tablet_manager.cc:1367] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 14:15:01.296133 31356 raft_consensus.cc:348] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:01.296372 31356 raft_consensus.cc:374] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:01.296495 31356 raft_consensus.cc:729] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:15:01.296741 31356 consensus_queue.cc:260] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:01.297276 31356 ts_tablet_manager.cc:1398] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:15:01.305523 31364 tablet_bootstrap.cc:492] T 3f6d55d89e55425183da73dbb3f4175f P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:15:01.305814 31364 ts_tablet_manager.cc:1367] T 3f6d55d89e55425183da73dbb3f4175f P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.005s
I20240502 14:15:01.307251 31364 raft_consensus.cc:348] T 3f6d55d89e55425183da73dbb3f4175f P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:01.307677 31364 raft_consensus.cc:374] T 3f6d55d89e55425183da73dbb3f4175f P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:01.307790 31364 raft_consensus.cc:729] T 3f6d55d89e55425183da73dbb3f4175f P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:15:01.308096 31364 consensus_queue.cc:260] T 3f6d55d89e55425183da73dbb3f4175f P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:01.309113 31364 ts_tablet_manager.cc:1398] T 3f6d55d89e55425183da73dbb3f4175f P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240502 14:15:01.309669 31355 tablet.cc:1722] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 68a3c2b17ed94e06aba66ed4178132ab: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:01.310349 31355 tablet.cc:1620] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 68a3c2b17ed94e06aba66ed4178132ab: MemRowSet was empty: no flush needed.
I20240502 14:15:01.338304 30926 raft_consensus.cc:482] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:01.338588 30926 raft_consensus.cc:504] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:01.339565 30926 leader_election.cc:288] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:15:01.341044 29176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d28bfbd29c94d2ebbedbae53231bf1e" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:15:01.341423 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d28bfbd29c94d2ebbedbae53231bf1e" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:15:01.341840 29309 raft_consensus.cc:2454] T 9d28bfbd29c94d2ebbedbae53231bf1e P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 0.
W20240502 14:15:01.342418 26493 leader_election.cc:341] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:01.342919 26493 leader_election.cc:302] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a, c1100e16b3b9495ca61d4f95e89b6a89; no voters: 1cb67158133e4648a6eceba32d4dbba3
I20240502 14:15:01.343518 30926 raft_consensus.cc:2781] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:01.343711 30926 raft_consensus.cc:482] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:01.343809 30926 raft_consensus.cc:3037] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:01.346918 30926 raft_consensus.cc:504] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:01.348291 29176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d28bfbd29c94d2ebbedbae53231bf1e" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
W20240502 14:15:01.349133 26493 leader_election.cc:341] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:01.350265 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d28bfbd29c94d2ebbedbae53231bf1e" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
I20240502 14:15:01.350492 29309 raft_consensus.cc:3037] T 9d28bfbd29c94d2ebbedbae53231bf1e P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:01.354066 29309 raft_consensus.cc:2454] T 9d28bfbd29c94d2ebbedbae53231bf1e P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:15:01.354547 26493 leader_election.cc:302] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a, c1100e16b3b9495ca61d4f95e89b6a89; no voters: 1cb67158133e4648a6eceba32d4dbba3
I20240502 14:15:01.354851 31095 raft_consensus.cc:2781] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:01.354902 30926 leader_election.cc:288] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Requested vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:15:01.355983 31366 tablet_bootstrap.cc:492] T be1c8c53ff6843a9bf07d006170f1f57 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:15:01.361578 31302 tablet_bootstrap.cc:492] T 9d28bfbd29c94d2ebbedbae53231bf1e P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:15:01.361860 31302 ts_tablet_manager.cc:1367] T 9d28bfbd29c94d2ebbedbae53231bf1e P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.180s user 0.045s sys 0.080s
I20240502 14:15:01.362879 31302 raft_consensus.cc:348] T 9d28bfbd29c94d2ebbedbae53231bf1e P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:01.363197 31302 raft_consensus.cc:374] T 9d28bfbd29c94d2ebbedbae53231bf1e P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:01.363327 31302 raft_consensus.cc:729] T 9d28bfbd29c94d2ebbedbae53231bf1e P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:15:01.363667 31302 consensus_queue.cc:260] T 9d28bfbd29c94d2ebbedbae53231bf1e P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:01.364362 31302 ts_tablet_manager.cc:1398] T 9d28bfbd29c94d2ebbedbae53231bf1e P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:15:01.365197 31095 raft_consensus.cc:686] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Becoming Leader. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Running, Role: LEADER
I20240502 14:15:01.365660 31095 consensus_queue.cc:237] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:01.366070 31297 tablet_bootstrap.cc:492] T 3f6d55d89e55425183da73dbb3f4175f P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:15:01.366271 31297 ts_tablet_manager.cc:1367] T 3f6d55d89e55425183da73dbb3f4175f P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.206s user 0.000s sys 0.010s
I20240502 14:15:01.367290 31297 raft_consensus.cc:348] T 3f6d55d89e55425183da73dbb3f4175f P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:01.367543 31297 raft_consensus.cc:374] T 3f6d55d89e55425183da73dbb3f4175f P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:01.367635 31297 raft_consensus.cc:729] T 3f6d55d89e55425183da73dbb3f4175f P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:15:01.367960 31297 consensus_queue.cc:260] T 3f6d55d89e55425183da73dbb3f4175f P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:01.368639 31297 ts_tablet_manager.cc:1398] T 3f6d55d89e55425183da73dbb3f4175f P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:15:01.368990 31297 tablet_bootstrap.cc:492] T be1c8c53ff6843a9bf07d006170f1f57 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:15:01.370421 31297 tablet_bootstrap.cc:654] T be1c8c53ff6843a9bf07d006170f1f57 P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:01.380261 31366 tablet_bootstrap.cc:654] T be1c8c53ff6843a9bf07d006170f1f57 P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:01.385622 26276 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:15:01.393342 26276 catalog_manager.cc:5484] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a reported cstate change: term changed from 0 to 1, leader changed from <none> to ba187e38a2754d60a12de9ac4856430a (127.19.94.194). New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:15:01.397555 31352 tablet.cc:1722] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 1cb67158133e4648a6eceba32d4dbba3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:01.400586 31355 tablet.cc:1722] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 68a3c2b17ed94e06aba66ed4178132ab: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:01.405292 31353 tablet.cc:1722] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:01.405479 31297 tablet_bootstrap.cc:492] T be1c8c53ff6843a9bf07d006170f1f57 P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:15:01.412214 26276 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table_new_name [id=c0ff6e6a8c224160b6c0977820de56f9] alter complete (version 3)
I20240502 14:15:01.405645 31297 ts_tablet_manager.cc:1367] T be1c8c53ff6843a9bf07d006170f1f57 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.037s user 0.001s sys 0.002s
I20240502 14:15:01.418130 31297 raft_consensus.cc:348] T be1c8c53ff6843a9bf07d006170f1f57 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:01.418380 31297 raft_consensus.cc:374] T be1c8c53ff6843a9bf07d006170f1f57 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:01.418484 31297 raft_consensus.cc:729] T be1c8c53ff6843a9bf07d006170f1f57 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:15:01.419138 31297 consensus_queue.cc:260] T be1c8c53ff6843a9bf07d006170f1f57 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:01.421201 31297 ts_tablet_manager.cc:1398] T be1c8c53ff6843a9bf07d006170f1f57 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 14:15:01.422801 31366 tablet_bootstrap.cc:492] T be1c8c53ff6843a9bf07d006170f1f57 P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:15:01.423000 31366 ts_tablet_manager.cc:1367] T be1c8c53ff6843a9bf07d006170f1f57 P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.067s user 0.004s sys 0.000s
I20240502 14:15:01.423947 31366 raft_consensus.cc:348] T be1c8c53ff6843a9bf07d006170f1f57 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:01.424182 31366 raft_consensus.cc:374] T be1c8c53ff6843a9bf07d006170f1f57 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:01.424351 31366 raft_consensus.cc:729] T be1c8c53ff6843a9bf07d006170f1f57 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:15:01.424698 31366 consensus_queue.cc:260] T be1c8c53ff6843a9bf07d006170f1f57 P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:01.425377 31366 ts_tablet_manager.cc:1398] T be1c8c53ff6843a9bf07d006170f1f57 P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 14:15:01.435089 31217 raft_consensus.cc:482] T 3f6d55d89e55425183da73dbb3f4175f P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:01.435578 31217 raft_consensus.cc:504] T 3f6d55d89e55425183da73dbb3f4175f P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:01.435755 31357 raft_consensus.cc:482] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:01.436187 31357 raft_consensus.cc:504] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:01.437361 31357 leader_election.cc:288] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:15:01.438373 29573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be1c8c53ff6843a9bf07d006170f1f57" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86" is_pre_election: true
I20240502 14:15:01.438354 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be1c8c53ff6843a9bf07d006170f1f57" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:15:01.438620 29573 raft_consensus.cc:2454] T be1c8c53ff6843a9bf07d006170f1f57 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 0.
I20240502 14:15:01.438637 29443 raft_consensus.cc:2454] T be1c8c53ff6843a9bf07d006170f1f57 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 0.
I20240502 14:15:01.439538 31217 leader_election.cc:288] T 3f6d55d89e55425183da73dbb3f4175f P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:15:01.440037 26755 leader_election.cc:302] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:15:01.440171 29441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6d55d89e55425183da73dbb3f4175f" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:15:01.440376 29441 raft_consensus.cc:2454] T 3f6d55d89e55425183da73dbb3f4175f P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 0.
I20240502 14:15:01.440841 31357 raft_consensus.cc:2781] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:01.440878 29245 leader_election.cc:302] T 3f6d55d89e55425183da73dbb3f4175f P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:15:01.440953 31357 raft_consensus.cc:482] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:01.441049 31357 raft_consensus.cc:3037] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:01.441378 31217 raft_consensus.cc:2781] T 3f6d55d89e55425183da73dbb3f4175f P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:01.441489 31217 raft_consensus.cc:482] T 3f6d55d89e55425183da73dbb3f4175f P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:01.441560 31217 raft_consensus.cc:3037] T 3f6d55d89e55425183da73dbb3f4175f P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:01.443028 26557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6d55d89e55425183da73dbb3f4175f" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:15:01.443303 26557 raft_consensus.cc:2454] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 0.
I20240502 14:15:01.444521 31357 raft_consensus.cc:504] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:01.445569 29441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be1c8c53ff6843a9bf07d006170f1f57" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
I20240502 14:15:01.445823 29441 raft_consensus.cc:3037] T be1c8c53ff6843a9bf07d006170f1f57 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:01.449131 29441 raft_consensus.cc:2454] T be1c8c53ff6843a9bf07d006170f1f57 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
I20240502 14:15:01.449793 26755 leader_election.cc:302] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:15:01.450243 31192 raft_consensus.cc:2781] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:01.450379 31192 raft_consensus.cc:686] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Becoming Leader. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Running, Role: LEADER
I20240502 14:15:01.450678 31192 consensus_queue.cc:237] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:01.452080 31357 leader_election.cc:288] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Requested vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:15:01.454231 29573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be1c8c53ff6843a9bf07d006170f1f57" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86"
I20240502 14:15:01.454514 29573 raft_consensus.cc:3037] T be1c8c53ff6843a9bf07d006170f1f57 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:01.458701 31217 raft_consensus.cc:504] T 3f6d55d89e55425183da73dbb3f4175f P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:01.458818 29573 raft_consensus.cc:2454] T be1c8c53ff6843a9bf07d006170f1f57 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
I20240502 14:15:01.460157 29441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6d55d89e55425183da73dbb3f4175f" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
I20240502 14:15:01.460620 29441 raft_consensus.cc:3037] T 3f6d55d89e55425183da73dbb3f4175f P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:01.471129 26557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6d55d89e55425183da73dbb3f4175f" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:15:01.471390 26557 raft_consensus.cc:3037] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:01.474483 26557 raft_consensus.cc:2454] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 1.
I20240502 14:15:01.475119 29244 leader_election.cc:302] T 3f6d55d89e55425183da73dbb3f4175f P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:15:01.484143 31217 leader_election.cc:288] T 3f6d55d89e55425183da73dbb3f4175f P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 election: Requested vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:15:01.487143 31217 raft_consensus.cc:2781] T 3f6d55d89e55425183da73dbb3f4175f P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:01.489655 29441 raft_consensus.cc:2454] T 3f6d55d89e55425183da73dbb3f4175f P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 1.
I20240502 14:15:01.489974 31217 raft_consensus.cc:686] T 3f6d55d89e55425183da73dbb3f4175f P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 LEADER]: Becoming Leader. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Running, Role: LEADER
I20240502 14:15:01.490334 31217 consensus_queue.cc:237] T 3f6d55d89e55425183da73dbb3f4175f P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
W20240502 14:15:01.494861 26624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.193:37985, user_credentials={real_user=slave}} blocked reactor thread for 55929.9us
I20240502 14:15:01.495257 26283 catalog_manager.cc:5484] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196). New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:01.496757 26276 catalog_manager.cc:5484] T 3f6d55d89e55425183da73dbb3f4175f P c1100e16b3b9495ca61d4f95e89b6a89 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198). New cstate: current_term: 1 leader_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:01.531021 26276 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:01.531360 26276 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:01.536280 26276 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet 92d6ce3fbd1c4cbb9caec04f7eb593e5
I20240502 14:15:01.537289 29156 tablet_service.cc:1514] Processing DeleteTablet for tablet 92d6ce3fbd1c4cbb9caec04f7eb593e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:01 UTC) from {username='slave'} at 127.0.0.1:55202
I20240502 14:15:01.537437 29423 tablet_service.cc:1514] Processing DeleteTablet for tablet 92d6ce3fbd1c4cbb9caec04f7eb593e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:01 UTC) from {username='slave'} at 127.0.0.1:40652
I20240502 14:15:01.538259 29289 tablet_service.cc:1514] Processing DeleteTablet for tablet 92d6ce3fbd1c4cbb9caec04f7eb593e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:01 UTC) from {username='slave'} at 127.0.0.1:59160
I20240502 14:15:01.579119 31391 tablet_replica.cc:330] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 1cb67158133e4648a6eceba32d4dbba3: stopping tablet replica
I20240502 14:15:01.579794 31391 raft_consensus.cc:2229] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:15:01.580458 31391 raft_consensus.cc:2258] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:01.589983 31391 ts_tablet_manager.cc:1875] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 1cb67158133e4648a6eceba32d4dbba3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:01.590775 31393 tablet_replica.cc:330] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P c1100e16b3b9495ca61d4f95e89b6a89: stopping tablet replica
I20240502 14:15:01.591003 31393 raft_consensus.cc:2229] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:01.591238 31393 raft_consensus.cc:2258] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:01.593024 31393 ts_tablet_manager.cc:1875] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:01.599690 31391 ts_tablet_manager.cc:1888] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 1cb67158133e4648a6eceba32d4dbba3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:01.599763 31392 tablet_replica.cc:330] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 68a3c2b17ed94e06aba66ed4178132ab: stopping tablet replica
I20240502 14:15:01.599889 31391 log.cc:1196] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 1cb67158133e4648a6eceba32d4dbba3: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal/wals/92d6ce3fbd1c4cbb9caec04f7eb593e5
I20240502 14:15:01.600026 31392 raft_consensus.cc:2229] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:01.602069 31391 ts_tablet_manager.cc:1909] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 1cb67158133e4648a6eceba32d4dbba3: Deleting consensus metadata
I20240502 14:15:01.603230 26271 catalog_manager.cc:4830] TS 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): tablet 92d6ce3fbd1c4cbb9caec04f7eb593e5 (table rebalancer_dynamic_table_new_name [id=c0ff6e6a8c224160b6c0977820de56f9]) successfully deleted
I20240502 14:15:01.600339 31392 raft_consensus.cc:2258] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:01.605592 31392 ts_tablet_manager.cc:1875] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 68a3c2b17ed94e06aba66ed4178132ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:01.611229 31392 ts_tablet_manager.cc:1888] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 68a3c2b17ed94e06aba66ed4178132ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:01.611485 31392 log.cc:1196] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 68a3c2b17ed94e06aba66ed4178132ab: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal/wals/92d6ce3fbd1c4cbb9caec04f7eb593e5
I20240502 14:15:01.612075 31392 ts_tablet_manager.cc:1909] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P 68a3c2b17ed94e06aba66ed4178132ab: Deleting consensus metadata
I20240502 14:15:01.612205 31393 ts_tablet_manager.cc:1888] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P c1100e16b3b9495ca61d4f95e89b6a89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:01.612426 31393 log.cc:1196] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal/wals/92d6ce3fbd1c4cbb9caec04f7eb593e5
I20240502 14:15:01.613677 31393 ts_tablet_manager.cc:1909] T 92d6ce3fbd1c4cbb9caec04f7eb593e5 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting consensus metadata
I20240502 14:15:01.614768 26272 catalog_manager.cc:4830] TS 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): tablet 92d6ce3fbd1c4cbb9caec04f7eb593e5 (table rebalancer_dynamic_table_new_name [id=c0ff6e6a8c224160b6c0977820de56f9]) successfully deleted
I20240502 14:15:01.614985 26271 catalog_manager.cc:4830] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): tablet 92d6ce3fbd1c4cbb9caec04f7eb593e5 (table rebalancer_dynamic_table_new_name [id=c0ff6e6a8c224160b6c0977820de56f9]) successfully deleted
I20240502 14:15:01.639575 26276 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:15:01.646564 26427 consensus_queue.cc:237] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:01.649410 29553 tablet_service.cc:1467] Processing CreateTablet for tablet 5d6936fc47ab4eada2b4311bc0b8257f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=98bc01f98f5f427e893dd681fe235c24]), partition=
I20240502 14:15:01.649453 29423 tablet_service.cc:1467] Processing CreateTablet for tablet 5d6936fc47ab4eada2b4311bc0b8257f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=98bc01f98f5f427e893dd681fe235c24]), partition=
I20240502 14:15:01.649876 29553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d6936fc47ab4eada2b4311bc0b8257f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:01.649989 29423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d6936fc47ab4eada2b4311bc0b8257f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:01.652812 29156 tablet_service.cc:1467] Processing CreateTablet for tablet 5d6936fc47ab4eada2b4311bc0b8257f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=98bc01f98f5f427e893dd681fe235c24]), partition=
I20240502 14:15:01.653206 29156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d6936fc47ab4eada2b4311bc0b8257f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:01.656755 31297 tablet_bootstrap.cc:492] T 5d6936fc47ab4eada2b4311bc0b8257f P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:15:01.658470 31297 tablet_bootstrap.cc:654] T 5d6936fc47ab4eada2b4311bc0b8257f P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:01.659138 31302 tablet_bootstrap.cc:492] T 5d6936fc47ab4eada2b4311bc0b8257f P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:15:01.660802 31302 tablet_bootstrap.cc:654] T 5d6936fc47ab4eada2b4311bc0b8257f P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:01.662433 31366 tablet_bootstrap.cc:492] T 5d6936fc47ab4eada2b4311bc0b8257f P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:15:01.668371 31366 tablet_bootstrap.cc:654] T 5d6936fc47ab4eada2b4311bc0b8257f P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:01.671186 29176 raft_consensus.cc:1261] T 0fc2e6c74dc34a45ae1ec46608642880 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:01.671286 26557 raft_consensus.cc:1261] T 0fc2e6c74dc34a45ae1ec46608642880 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:01.672154 31173 consensus_queue.cc:1035] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:01.672901 31297 tablet_bootstrap.cc:492] T 5d6936fc47ab4eada2b4311bc0b8257f P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:15:01.672807 31173 consensus_queue.cc:1035] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:01.673091 31297 ts_tablet_manager.cc:1367] T 5d6936fc47ab4eada2b4311bc0b8257f P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.003s
I20240502 14:15:01.673775 31297 raft_consensus.cc:348] T 5d6936fc47ab4eada2b4311bc0b8257f P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:01.673979 31297 raft_consensus.cc:374] T 5d6936fc47ab4eada2b4311bc0b8257f P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:01.674063 31297 raft_consensus.cc:729] T 5d6936fc47ab4eada2b4311bc0b8257f P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:15:01.674311 31297 consensus_queue.cc:260] T 5d6936fc47ab4eada2b4311bc0b8257f P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:01.674963 31297 ts_tablet_manager.cc:1398] T 5d6936fc47ab4eada2b4311bc0b8257f P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:15:01.686744 31302 tablet_bootstrap.cc:492] T 5d6936fc47ab4eada2b4311bc0b8257f P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:15:01.686589 31403 raft_consensus.cc:2932] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:01.687249 26557 raft_consensus.cc:2932] T 0fc2e6c74dc34a45ae1ec46608642880 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:01.687848 31302 ts_tablet_manager.cc:1367] T 5d6936fc47ab4eada2b4311bc0b8257f P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.028s user 0.010s sys 0.003s
I20240502 14:15:01.689200 31302 raft_consensus.cc:348] T 5d6936fc47ab4eada2b4311bc0b8257f P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:01.691008 31302 raft_consensus.cc:374] T 5d6936fc47ab4eada2b4311bc0b8257f P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:01.694546 26283 catalog_manager.cc:5484] T 0fc2e6c74dc34a45ae1ec46608642880 P ba187e38a2754d60a12de9ac4856430a reported cstate change: config changed from index 4 to 5, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:01.697775 31302 raft_consensus.cc:729] T 5d6936fc47ab4eada2b4311bc0b8257f P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:15:01.698511 31302 consensus_queue.cc:260] T 5d6936fc47ab4eada2b4311bc0b8257f P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:01.700542 26668 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fc2e6c74dc34a45ae1ec46608642880 with delete_type TABLET_DATA_TOMBSTONED (TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34150
I20240502 14:15:01.706982 31407 tablet_replica.cc:330] T 0fc2e6c74dc34a45ae1ec46608642880 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: stopping tablet replica
I20240502 14:15:01.708021 31407 raft_consensus.cc:2229] T 0fc2e6c74dc34a45ae1ec46608642880 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:01.708297 31407 raft_consensus.cc:2258] T 0fc2e6c74dc34a45ae1ec46608642880 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:01.709605 31407 ts_tablet_manager.cc:1875] T 0fc2e6c74dc34a45ae1ec46608642880 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:01.721351 31407 ts_tablet_manager.cc:1888] T 0fc2e6c74dc34a45ae1ec46608642880 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:15:01.721668 31407 log.cc:1196] T 0fc2e6c74dc34a45ae1ec46608642880 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/wals/0fc2e6c74dc34a45ae1ec46608642880
I20240502 14:15:01.721930 31181 raft_consensus.cc:482] T 9d28bfbd29c94d2ebbedbae53231bf1e P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:01.722173 31181 raft_consensus.cc:504] T 9d28bfbd29c94d2ebbedbae53231bf1e P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:01.723238 31181 leader_election.cc:288] T 9d28bfbd29c94d2ebbedbae53231bf1e P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:15:01.724217 31302 ts_tablet_manager.cc:1398] T 5d6936fc47ab4eada2b4311bc0b8257f P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.036s user 0.002s sys 0.000s
I20240502 14:15:01.725135 29307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d28bfbd29c94d2ebbedbae53231bf1e" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:15:01.725493 29307 raft_consensus.cc:2379] T 9d28bfbd29c94d2ebbedbae53231bf1e P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb67158133e4648a6eceba32d4dbba3 in current term 1: Already voted for candidate ba187e38a2754d60a12de9ac4856430a in this term.
I20240502 14:15:01.727322 26557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d28bfbd29c94d2ebbedbae53231bf1e" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:15:01.729091 29111 leader_election.cc:302] T 9d28bfbd29c94d2ebbedbae53231bf1e P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3; no voters: ba187e38a2754d60a12de9ac4856430a, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:15:01.729523 30953 raft_consensus.cc:3037] T 9d28bfbd29c94d2ebbedbae53231bf1e P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:01.732334 30953 raft_consensus.cc:2726] T 9d28bfbd29c94d2ebbedbae53231bf1e P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:15:01.733590 29176 raft_consensus.cc:2932] T 0fc2e6c74dc34a45ae1ec46608642880 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:01.735718 26272 catalog_manager.cc:4830] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): tablet 0fc2e6c74dc34a45ae1ec46608642880 (table rebalance_test_table_1 [id=d7c1bbb69be74fd3a8e809dd8649698d]) successfully deleted
I20240502 14:15:01.738371 31366 tablet_bootstrap.cc:492] T 5d6936fc47ab4eada2b4311bc0b8257f P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:15:01.738720 31366 ts_tablet_manager.cc:1367] T 5d6936fc47ab4eada2b4311bc0b8257f P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.076s user 0.014s sys 0.029s
I20240502 14:15:01.739966 31366 raft_consensus.cc:348] T 5d6936fc47ab4eada2b4311bc0b8257f P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:01.740470 31366 raft_consensus.cc:374] T 5d6936fc47ab4eada2b4311bc0b8257f P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:01.740563 31366 raft_consensus.cc:729] T 5d6936fc47ab4eada2b4311bc0b8257f P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:15:01.740810 31366 consensus_queue.cc:260] T 5d6936fc47ab4eada2b4311bc0b8257f P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:01.741585 31366 ts_tablet_manager.cc:1398] T 5d6936fc47ab4eada2b4311bc0b8257f P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:15:01.764572 31363 raft_consensus.cc:482] T 5d6936fc47ab4eada2b4311bc0b8257f P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:01.764839 31363 raft_consensus.cc:504] T 5d6936fc47ab4eada2b4311bc0b8257f P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:01.765933 31363 leader_election.cc:288] T 5d6936fc47ab4eada2b4311bc0b8257f P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:15:01.766361 29176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d6936fc47ab4eada2b4311bc0b8257f" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:15:01.766665 29176 raft_consensus.cc:2454] T 5d6936fc47ab4eada2b4311bc0b8257f P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 0.
I20240502 14:15:01.766810 29573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d6936fc47ab4eada2b4311bc0b8257f" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86" is_pre_election: true
I20240502 14:15:01.767174 29573 raft_consensus.cc:2454] T 5d6936fc47ab4eada2b4311bc0b8257f P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 0.
I20240502 14:15:01.767313 29378 leader_election.cc:302] T 5d6936fc47ab4eada2b4311bc0b8257f P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 68a3c2b17ed94e06aba66ed4178132ab; no voters:
I20240502 14:15:01.767768 31363 raft_consensus.cc:2781] T 5d6936fc47ab4eada2b4311bc0b8257f P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:01.767887 31363 raft_consensus.cc:482] T 5d6936fc47ab4eada2b4311bc0b8257f P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:01.767969 31363 raft_consensus.cc:3037] T 5d6936fc47ab4eada2b4311bc0b8257f P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:01.773196 31363 raft_consensus.cc:504] T 5d6936fc47ab4eada2b4311bc0b8257f P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:01.775419 29176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d6936fc47ab4eada2b4311bc0b8257f" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:15:01.775658 29176 raft_consensus.cc:3037] T 5d6936fc47ab4eada2b4311bc0b8257f P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:01.779299 29176 raft_consensus.cc:2454] T 5d6936fc47ab4eada2b4311bc0b8257f P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 1.
I20240502 14:15:01.779867 29378 leader_election.cc:302] T 5d6936fc47ab4eada2b4311bc0b8257f P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 68a3c2b17ed94e06aba66ed4178132ab; no voters:
I20240502 14:15:01.780817 31419 raft_consensus.cc:2781] T 5d6936fc47ab4eada2b4311bc0b8257f P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:01.781033 31363 leader_election.cc:288] T 5d6936fc47ab4eada2b4311bc0b8257f P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 election: Requested vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:15:01.781590 31419 raft_consensus.cc:686] T 5d6936fc47ab4eada2b4311bc0b8257f P 68a3c2b17ed94e06aba66ed4178132ab [term 1 LEADER]: Becoming Leader. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Running, Role: LEADER
I20240502 14:15:01.781966 31419 consensus_queue.cc:237] T 5d6936fc47ab4eada2b4311bc0b8257f P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:01.782553 29573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d6936fc47ab4eada2b4311bc0b8257f" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86"
I20240502 14:15:01.782816 29573 raft_consensus.cc:3037] T 5d6936fc47ab4eada2b4311bc0b8257f P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:01.785039 29573 raft_consensus.cc:2454] T 5d6936fc47ab4eada2b4311bc0b8257f P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 1.
I20240502 14:15:01.786223 26276 catalog_manager.cc:5484] T 5d6936fc47ab4eada2b4311bc0b8257f P 68a3c2b17ed94e06aba66ed4178132ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199). New cstate: current_term: 1 leader_uuid: "68a3c2b17ed94e06aba66ed4178132ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:15:01.852087 31095 consensus_queue.cc:1035] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:01.879244 31095 consensus_queue.cc:1035] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:01.913555 31374 consensus_queue.cc:1035] T 3f6d55d89e55425183da73dbb3f4175f P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:01.919373 31378 consensus_queue.cc:1035] T 3f6d55d89e55425183da73dbb3f4175f P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:01.925494 26278 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:15:01.946537 29176 raft_consensus.cc:1261] T 5d6936fc47ab4eada2b4311bc0b8257f P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:01.947129 29573 raft_consensus.cc:1261] T 5d6936fc47ab4eada2b4311bc0b8257f P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:01.948079 31420 consensus_queue.cc:1035] T 5d6936fc47ab4eada2b4311bc0b8257f P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:01.948727 31420 consensus_queue.cc:1035] T 5d6936fc47ab4eada2b4311bc0b8257f P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:01.972429 26283 catalog_manager.cc:5576] TS 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715) does not have the latest schema for tablet 5d6936fc47ab4eada2b4311bc0b8257f (table rebalancer_dynamic_table [id=98bc01f98f5f427e893dd681fe235c24]). Expected version 1 got 0
I20240502 14:15:01.972821 31455 tablet.cc:1722] T 5d6936fc47ab4eada2b4311bc0b8257f P 68a3c2b17ed94e06aba66ed4178132ab: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:01.973485 31455 tablet.cc:1620] T 5d6936fc47ab4eada2b4311bc0b8257f P 68a3c2b17ed94e06aba66ed4178132ab: MemRowSet was empty: no flush needed.
I20240502 14:15:01.985677 26285 catalog_manager.cc:5576] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097) does not have the latest schema for tablet 5d6936fc47ab4eada2b4311bc0b8257f (table rebalancer_dynamic_table [id=98bc01f98f5f427e893dd681fe235c24]). Expected version 1 got 0
I20240502 14:15:01.993203 31458 tablet.cc:1722] T 5d6936fc47ab4eada2b4311bc0b8257f P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:01.995405 31458 tablet.cc:1620] T 5d6936fc47ab4eada2b4311bc0b8257f P 4092902b3e674857a0147f1b4f2cdb86: MemRowSet was empty: no flush needed.
I20240502 14:15:02.003223 26279 catalog_manager.cc:5576] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097) does not have the latest schema for tablet 5d6936fc47ab4eada2b4311bc0b8257f (table rebalancer_dynamic_table [id=98bc01f98f5f427e893dd681fe235c24]). Expected version 1 got 0
I20240502 14:15:02.013089 26276 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=98bc01f98f5f427e893dd681fe235c24] alter complete (version 1)
I20240502 14:15:02.016073 26276 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_extra_table_26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:02.028792 31357 consensus_queue.cc:1035] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:02.034619 29156 tablet_service.cc:1467] Processing CreateTablet for tablet 13136deb9046440f88bf3471ff42312c (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=88ec6ba556824ae1ac8b611f69608326]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:02.035080 26537 tablet_service.cc:1467] Processing CreateTablet for tablet 13136deb9046440f88bf3471ff42312c (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=88ec6ba556824ae1ac8b611f69608326]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:02.035230 29156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13136deb9046440f88bf3471ff42312c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:02.035264 29553 tablet_service.cc:1467] Processing CreateTablet for tablet 01b1a1d502874a3791af520982ecacc0 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=88ec6ba556824ae1ac8b611f69608326]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:02.035602 26537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13136deb9046440f88bf3471ff42312c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:02.035694 29553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01b1a1d502874a3791af520982ecacc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:02.042446 31366 tablet_bootstrap.cc:492] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:15:02.044023 29423 tablet_service.cc:1467] Processing CreateTablet for tablet 01b1a1d502874a3791af520982ecacc0 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=88ec6ba556824ae1ac8b611f69608326]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:02.044499 29423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01b1a1d502874a3791af520982ecacc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:02.044705 31366 tablet_bootstrap.cc:654] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:02.049333 29154 tablet_service.cc:1467] Processing CreateTablet for tablet 01b1a1d502874a3791af520982ecacc0 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=88ec6ba556824ae1ac8b611f69608326]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:02.049777 29154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01b1a1d502874a3791af520982ecacc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:02.052532 31192 consensus_queue.cc:1035] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:02.056694 31297 tablet_bootstrap.cc:492] T 01b1a1d502874a3791af520982ecacc0 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:15:02.058625 31297 tablet_bootstrap.cc:654] T 01b1a1d502874a3791af520982ecacc0 P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:02.061285 31302 tablet_bootstrap.cc:492] T 01b1a1d502874a3791af520982ecacc0 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:15:02.063635 31302 tablet_bootstrap.cc:654] T 01b1a1d502874a3791af520982ecacc0 P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:02.071923 26797 tablet_service.cc:1467] Processing CreateTablet for tablet f6561a18519245f9a99690259c9c295e (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=88ec6ba556824ae1ac8b611f69608326]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:02.072031 29156 tablet_service.cc:1467] Processing CreateTablet for tablet f6561a18519245f9a99690259c9c295e (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=88ec6ba556824ae1ac8b611f69608326]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
W20240502 14:15:02.072420 26625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.196:32843, user_credentials={real_user=slave}} blocked reactor thread for 53567.6us
I20240502 14:15:02.072532 29156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6561a18519245f9a99690259c9c295e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:02.072794 26797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6561a18519245f9a99690259c9c295e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:02.081017 31366 tablet_bootstrap.cc:492] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:15:02.081202 31366 ts_tablet_manager.cc:1367] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.039s user 0.008s sys 0.004s
I20240502 14:15:02.085976 31366 raft_consensus.cc:348] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:02.086263 31366 raft_consensus.cc:374] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:02.086369 31366 raft_consensus.cc:729] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:15:02.087780 31366 consensus_queue.cc:260] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:02.088543 31366 ts_tablet_manager.cc:1398] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20240502 14:15:02.098407 29553 tablet_service.cc:1467] Processing CreateTablet for tablet f6561a18519245f9a99690259c9c295e (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=88ec6ba556824ae1ac8b611f69608326]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:02.099100 29553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6561a18519245f9a99690259c9c295e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:02.149616 26285 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:15:02.151856 29289 tablet_service.cc:1467] Processing CreateTablet for tablet 13136deb9046440f88bf3471ff42312c (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=88ec6ba556824ae1ac8b611f69608326]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:02.152369 29289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13136deb9046440f88bf3471ff42312c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:02.153916 31468 tablet_bootstrap.cc:492] T f6561a18519245f9a99690259c9c295e P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:15:02.155690 31468 tablet_bootstrap.cc:654] T f6561a18519245f9a99690259c9c295e P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:02.157989 31302 tablet_bootstrap.cc:492] T 01b1a1d502874a3791af520982ecacc0 P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:15:02.158291 31302 ts_tablet_manager.cc:1367] T 01b1a1d502874a3791af520982ecacc0 P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.097s user 0.006s sys 0.006s
I20240502 14:15:02.160972 31302 raft_consensus.cc:348] T 01b1a1d502874a3791af520982ecacc0 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:02.161345 31302 raft_consensus.cc:374] T 01b1a1d502874a3791af520982ecacc0 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:02.162724 31297 tablet_bootstrap.cc:492] T 01b1a1d502874a3791af520982ecacc0 P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:15:02.162953 31297 ts_tablet_manager.cc:1367] T 01b1a1d502874a3791af520982ecacc0 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.106s user 0.003s sys 0.008s
I20240502 14:15:02.168493 31466 tablet_bootstrap.cc:492] T 13136deb9046440f88bf3471ff42312c P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:15:02.171411 31466 tablet_bootstrap.cc:654] T 13136deb9046440f88bf3471ff42312c P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:02.176826 31457 tablet.cc:1722] T 5d6936fc47ab4eada2b4311bc0b8257f P 1cb67158133e4648a6eceba32d4dbba3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:02.177544 31457 tablet.cc:1620] T 5d6936fc47ab4eada2b4311bc0b8257f P 1cb67158133e4648a6eceba32d4dbba3: MemRowSet was empty: no flush needed.
I20240502 14:15:02.180277 26277 catalog_manager.cc:5576] TS 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715) does not have the latest schema for tablet 5d6936fc47ab4eada2b4311bc0b8257f (table rebalancer_dynamic_table [id=98bc01f98f5f427e893dd681fe235c24]). Expected version 2 got 0
I20240502 14:15:02.164287 31297 raft_consensus.cc:348] T 01b1a1d502874a3791af520982ecacc0 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:02.182065 31297 raft_consensus.cc:374] T 01b1a1d502874a3791af520982ecacc0 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:02.182204 31297 raft_consensus.cc:729] T 01b1a1d502874a3791af520982ecacc0 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:15:02.182487 31297 consensus_queue.cc:260] T 01b1a1d502874a3791af520982ecacc0 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:02.183087 31297 ts_tablet_manager.cc:1398] T 01b1a1d502874a3791af520982ecacc0 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20240502 14:15:02.185503 31302 raft_consensus.cc:729] T 01b1a1d502874a3791af520982ecacc0 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:15:02.185994 31302 consensus_queue.cc:260] T 01b1a1d502874a3791af520982ecacc0 P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:02.186249 31458 tablet.cc:1722] T 5d6936fc47ab4eada2b4311bc0b8257f P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:02.186805 31458 tablet.cc:1620] T 5d6936fc47ab4eada2b4311bc0b8257f P 4092902b3e674857a0147f1b4f2cdb86: MemRowSet was empty: no flush needed.
I20240502 14:15:02.187995 31302 ts_tablet_manager.cc:1398] T 01b1a1d502874a3791af520982ecacc0 P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.029s user 0.001s sys 0.002s
I20240502 14:15:02.188433 31302 tablet_bootstrap.cc:492] T 13136deb9046440f88bf3471ff42312c P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:15:02.190436 31302 tablet_bootstrap.cc:654] T 13136deb9046440f88bf3471ff42312c P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:02.192322 31366 tablet_bootstrap.cc:492] T f6561a18519245f9a99690259c9c295e P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:15:02.193888 31366 tablet_bootstrap.cc:654] T f6561a18519245f9a99690259c9c295e P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:02.195318 31455 tablet.cc:1722] T 5d6936fc47ab4eada2b4311bc0b8257f P 68a3c2b17ed94e06aba66ed4178132ab: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:02.196123 30448 raft_consensus.cc:482] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:02.196287 30448 raft_consensus.cc:504] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:02.197120 30448 leader_election.cc:288] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:15:02.197831 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b1a1d502874a3791af520982ecacc0" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:15:02.198074 29443 raft_consensus.cc:2454] T 01b1a1d502874a3791af520982ecacc0 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:15:02.198619 29511 leader_election.cc:302] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, 68a3c2b17ed94e06aba66ed4178132ab; no voters:
I20240502 14:15:02.198963 31366 tablet_bootstrap.cc:492] T f6561a18519245f9a99690259c9c295e P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:15:02.199573 31366 ts_tablet_manager.cc:1367] T f6561a18519245f9a99690259c9c295e P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 14:15:02.199884 30448 raft_consensus.cc:2781] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:02.200011 30448 raft_consensus.cc:482] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:02.200100 30448 raft_consensus.cc:3037] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:02.203285 26283 catalog_manager.cc:5576] TS 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715) does not have the latest schema for tablet 5d6936fc47ab4eada2b4311bc0b8257f (table rebalancer_dynamic_table [id=98bc01f98f5f427e893dd681fe235c24]). Expected version 2 got 1
I20240502 14:15:02.203825 31366 raft_consensus.cc:348] T f6561a18519245f9a99690259c9c295e P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:02.204139 31366 raft_consensus.cc:374] T f6561a18519245f9a99690259c9c295e P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:02.204259 31366 raft_consensus.cc:729] T f6561a18519245f9a99690259c9c295e P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:15:02.204584 31366 consensus_queue.cc:260] T f6561a18519245f9a99690259c9c295e P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:02.205829 31366 ts_tablet_manager.cc:1398] T f6561a18519245f9a99690259c9c295e P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 14:15:02.205863 31466 tablet_bootstrap.cc:492] T 13136deb9046440f88bf3471ff42312c P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:15:02.206032 31466 ts_tablet_manager.cc:1367] T 13136deb9046440f88bf3471ff42312c P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.038s user 0.002s sys 0.003s
I20240502 14:15:02.206364 29176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b1a1d502874a3791af520982ecacc0" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:15:02.206657 29176 raft_consensus.cc:2454] T 01b1a1d502874a3791af520982ecacc0 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:15:02.207648 30448 raft_consensus.cc:504] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:02.208954 29176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b1a1d502874a3791af520982ecacc0" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:15:02.209089 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b1a1d502874a3791af520982ecacc0" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
I20240502 14:15:02.209213 29176 raft_consensus.cc:3037] T 01b1a1d502874a3791af520982ecacc0 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:02.209343 29443 raft_consensus.cc:3037] T 01b1a1d502874a3791af520982ecacc0 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:02.209360 26279 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=98bc01f98f5f427e893dd681fe235c24] alter complete (version 2)
I20240502 14:15:02.209726 31466 raft_consensus.cc:348] T 13136deb9046440f88bf3471ff42312c P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:02.209940 31466 raft_consensus.cc:374] T 13136deb9046440f88bf3471ff42312c P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:02.210044 31466 raft_consensus.cc:729] T 13136deb9046440f88bf3471ff42312c P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:15:02.210305 31466 consensus_queue.cc:260] T 13136deb9046440f88bf3471ff42312c P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:02.211907 31302 tablet_bootstrap.cc:492] T 13136deb9046440f88bf3471ff42312c P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:15:02.213063 29176 raft_consensus.cc:2454] T 01b1a1d502874a3791af520982ecacc0 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:15:02.214051 31302 ts_tablet_manager.cc:1367] T 13136deb9046440f88bf3471ff42312c P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20240502 14:15:02.215534 31302 raft_consensus.cc:348] T 13136deb9046440f88bf3471ff42312c P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:02.215821 29443 raft_consensus.cc:2454] T 01b1a1d502874a3791af520982ecacc0 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:15:02.216503 29511 leader_election.cc:302] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, 68a3c2b17ed94e06aba66ed4178132ab; no voters:
I20240502 14:15:02.217198 31466 ts_tablet_manager.cc:1398] T 13136deb9046440f88bf3471ff42312c P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20240502 14:15:02.217437 30448 leader_election.cc:288] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Requested vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:15:02.217732 31203 raft_consensus.cc:2781] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:02.217898 31203 raft_consensus.cc:686] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Becoming Leader. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Running, Role: LEADER
I20240502 14:15:02.218156 31203 consensus_queue.cc:237] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:02.221608 31302 raft_consensus.cc:374] T 13136deb9046440f88bf3471ff42312c P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:02.221746 31302 raft_consensus.cc:729] T 13136deb9046440f88bf3471ff42312c P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:15:02.221987 31302 consensus_queue.cc:260] T 13136deb9046440f88bf3471ff42312c P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:02.222734 31302 ts_tablet_manager.cc:1398] T 13136deb9046440f88bf3471ff42312c P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
I20240502 14:15:02.224709 31455 tablet.cc:1620] T 5d6936fc47ab4eada2b4311bc0b8257f P 68a3c2b17ed94e06aba66ed4178132ab: MemRowSet was empty: no flush needed.
I20240502 14:15:02.233521 31457 tablet.cc:1722] T 5d6936fc47ab4eada2b4311bc0b8257f P 1cb67158133e4648a6eceba32d4dbba3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:02.234087 31457 tablet.cc:1620] T 5d6936fc47ab4eada2b4311bc0b8257f P 1cb67158133e4648a6eceba32d4dbba3: MemRowSet was empty: no flush needed.
I20240502 14:15:02.236194 31302 tablet_bootstrap.cc:492] T f6561a18519245f9a99690259c9c295e P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:15:02.238373 31302 tablet_bootstrap.cc:654] T f6561a18519245f9a99690259c9c295e P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:02.238008 26277 catalog_manager.cc:5484] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200). New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:02.257860 29156 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:02.298825 31468 tablet_bootstrap.cc:492] T f6561a18519245f9a99690259c9c295e P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:15:02.299108 31468 ts_tablet_manager.cc:1367] T f6561a18519245f9a99690259c9c295e P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.145s user 0.026s sys 0.011s
I20240502 14:15:02.300067 31468 raft_consensus.cc:348] T f6561a18519245f9a99690259c9c295e P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:02.300382 31468 raft_consensus.cc:374] T f6561a18519245f9a99690259c9c295e P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:02.300606 31468 raft_consensus.cc:729] T f6561a18519245f9a99690259c9c295e P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:15:02.301084 31468 consensus_queue.cc:260] T f6561a18519245f9a99690259c9c295e P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:02.302028 31468 ts_tablet_manager.cc:1398] T f6561a18519245f9a99690259c9c295e P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:15:02.304502 31469 tablet_bootstrap.cc:492] T 13136deb9046440f88bf3471ff42312c P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:15:02.311764 31302 tablet_bootstrap.cc:492] T f6561a18519245f9a99690259c9c295e P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:15:02.311971 31302 ts_tablet_manager.cc:1367] T f6561a18519245f9a99690259c9c295e P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.076s user 0.005s sys 0.012s
I20240502 14:15:02.312830 31302 raft_consensus.cc:348] T f6561a18519245f9a99690259c9c295e P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:02.313104 31302 raft_consensus.cc:374] T f6561a18519245f9a99690259c9c295e P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:02.313195 31302 raft_consensus.cc:729] T f6561a18519245f9a99690259c9c295e P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:15:02.313637 31302 consensus_queue.cc:260] T f6561a18519245f9a99690259c9c295e P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:02.317371 31302 ts_tablet_manager.cc:1398] T f6561a18519245f9a99690259c9c295e P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240502 14:15:02.336719 31469 tablet_bootstrap.cc:654] T 13136deb9046440f88bf3471ff42312c P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:02.383576 31469 tablet_bootstrap.cc:492] T 13136deb9046440f88bf3471ff42312c P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:15:02.383934 31469 ts_tablet_manager.cc:1367] T 13136deb9046440f88bf3471ff42312c P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.080s user 0.004s sys 0.004s
I20240502 14:15:02.384964 31469 raft_consensus.cc:348] T 13136deb9046440f88bf3471ff42312c P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:02.385290 31469 raft_consensus.cc:374] T 13136deb9046440f88bf3471ff42312c P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:02.385459 31469 raft_consensus.cc:729] T 13136deb9046440f88bf3471ff42312c P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:15:02.385792 31469 consensus_queue.cc:260] T 13136deb9046440f88bf3471ff42312c P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:02.387676 31469 ts_tablet_manager.cc:1398] T 13136deb9046440f88bf3471ff42312c P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:15:02.388648 29423 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:15:02.391625 26285 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:15:02.391793 26407 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:02.396334 29553 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:02.410986 31455 tablet.cc:1722] T 5d6936fc47ab4eada2b4311bc0b8257f P 68a3c2b17ed94e06aba66ed4178132ab: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:02.411936 31458 tablet.cc:1722] T 5d6936fc47ab4eada2b4311bc0b8257f P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:02.415307 26537 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:02.415690 30953 raft_consensus.cc:482] T f6561a18519245f9a99690259c9c295e P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:02.415937 30953 raft_consensus.cc:504] T f6561a18519245f9a99690259c9c295e P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:02.416854 30953 leader_election.cc:288] T f6561a18519245f9a99690259c9c295e P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:15:02.418323 26817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6561a18519245f9a99690259c9c295e" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:15:02.418699 26817 raft_consensus.cc:2454] T f6561a18519245f9a99690259c9c295e P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 0.
I20240502 14:15:02.434434 29112 leader_election.cc:302] T f6561a18519245f9a99690259c9c295e P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:15:02.435003 30953 raft_consensus.cc:2781] T f6561a18519245f9a99690259c9c295e P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:02.435137 30953 raft_consensus.cc:482] T f6561a18519245f9a99690259c9c295e P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:02.435235 30953 raft_consensus.cc:3037] T f6561a18519245f9a99690259c9c295e P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:02.438196 30953 raft_consensus.cc:504] T f6561a18519245f9a99690259c9c295e P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:02.438891 30953 leader_election.cc:288] T f6561a18519245f9a99690259c9c295e P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 election: Requested vote from peers 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:15:02.442165 29575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6561a18519245f9a99690259c9c295e" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86"
I20240502 14:15:02.442418 29575 raft_consensus.cc:3037] T f6561a18519245f9a99690259c9c295e P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:02.442682 26817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6561a18519245f9a99690259c9c295e" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:15:02.442943 26817 raft_consensus.cc:3037] T f6561a18519245f9a99690259c9c295e P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:02.443033 29573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6561a18519245f9a99690259c9c295e" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86" is_pre_election: true
I20240502 14:15:02.445194 26277 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table_new_name [id=98bc01f98f5f427e893dd681fe235c24] alter complete (version 3)
I20240502 14:15:02.445490 29575 raft_consensus.cc:2454] T f6561a18519245f9a99690259c9c295e P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 1.
I20240502 14:15:02.445789 26817 raft_consensus.cc:2454] T f6561a18519245f9a99690259c9c295e P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 1.
I20240502 14:15:02.446723 31457 tablet.cc:1722] T 5d6936fc47ab4eada2b4311bc0b8257f P 1cb67158133e4648a6eceba32d4dbba3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:02.448113 26797 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:02.448293 29112 leader_election.cc:302] T f6561a18519245f9a99690259c9c295e P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:15:02.452958 31181 raft_consensus.cc:2781] T f6561a18519245f9a99690259c9c295e P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:02.453109 31181 raft_consensus.cc:686] T f6561a18519245f9a99690259c9c295e P 1cb67158133e4648a6eceba32d4dbba3 [term 1 LEADER]: Becoming Leader. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Running, Role: LEADER
I20240502 14:15:02.453401 31181 consensus_queue.cc:237] T f6561a18519245f9a99690259c9c295e P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:02.484768 26285 catalog_manager.cc:5484] T f6561a18519245f9a99690259c9c295e P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197). New cstate: current_term: 1 leader_uuid: "1cb67158133e4648a6eceba32d4dbba3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:02.487339 31217 raft_consensus.cc:482] T 13136deb9046440f88bf3471ff42312c P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:02.487574 31217 raft_consensus.cc:504] T 13136deb9046440f88bf3471ff42312c P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:02.488240 31217 leader_election.cc:288] T 13136deb9046440f88bf3471ff42312c P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:15:02.490259 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13136deb9046440f88bf3471ff42312c" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:15:02.490407 29176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13136deb9046440f88bf3471ff42312c" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:15:02.490521 26558 raft_consensus.cc:2454] T 13136deb9046440f88bf3471ff42312c P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 0.
I20240502 14:15:02.490672 29176 raft_consensus.cc:2454] T 13136deb9046440f88bf3471ff42312c P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 0.
I20240502 14:15:02.494055 29244 leader_election.cc:302] T 13136deb9046440f88bf3471ff42312c P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:15:02.494472 31217 raft_consensus.cc:2781] T 13136deb9046440f88bf3471ff42312c P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:02.494581 31217 raft_consensus.cc:482] T 13136deb9046440f88bf3471ff42312c P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:02.494678 31217 raft_consensus.cc:3037] T 13136deb9046440f88bf3471ff42312c P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:02.497668 31217 raft_consensus.cc:504] T 13136deb9046440f88bf3471ff42312c P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:02.498564 31217 leader_election.cc:288] T 13136deb9046440f88bf3471ff42312c P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 election: Requested vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:15:02.501196 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13136deb9046440f88bf3471ff42312c" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:15:02.501561 26558 raft_consensus.cc:3037] T 13136deb9046440f88bf3471ff42312c P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:02.503594 29289 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:15:02.504273 29176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13136deb9046440f88bf3471ff42312c" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:15:02.504542 29176 raft_consensus.cc:3037] T 13136deb9046440f88bf3471ff42312c P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:02.505367 26558 raft_consensus.cc:2454] T 13136deb9046440f88bf3471ff42312c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 1.
I20240502 14:15:02.512696 29176 raft_consensus.cc:2454] T 13136deb9046440f88bf3471ff42312c P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 1.
I20240502 14:15:02.514271 29244 leader_election.cc:302] T 13136deb9046440f88bf3471ff42312c P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:15:02.515486 31217 raft_consensus.cc:2781] T 13136deb9046440f88bf3471ff42312c P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:02.515667 31217 raft_consensus.cc:686] T 13136deb9046440f88bf3471ff42312c P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 LEADER]: Becoming Leader. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Running, Role: LEADER
I20240502 14:15:02.515916 31217 consensus_queue.cc:237] T 13136deb9046440f88bf3471ff42312c P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:02.549401 26285 catalog_manager.cc:5484] T 13136deb9046440f88bf3471ff42312c P c1100e16b3b9495ca61d4f95e89b6a89 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198). New cstate: current_term: 1 leader_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:02.555871 26668 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:15:02.572283 26285 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:02.572575 26285 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:02.576692 26285 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet 5d6936fc47ab4eada2b4311bc0b8257f
I20240502 14:15:02.577550 29156 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d6936fc47ab4eada2b4311bc0b8257f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:02 UTC) from {username='slave'} at 127.0.0.1:55202
I20240502 14:15:02.580363 29423 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d6936fc47ab4eada2b4311bc0b8257f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:02 UTC) from {username='slave'} at 127.0.0.1:40652
I20240502 14:15:02.588356 29553 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d6936fc47ab4eada2b4311bc0b8257f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:02 UTC) from {username='slave'} at 127.0.0.1:36398
I20240502 14:15:02.590534 31501 tablet_replica.cc:330] T 5d6936fc47ab4eada2b4311bc0b8257f P 68a3c2b17ed94e06aba66ed4178132ab: stopping tablet replica
I20240502 14:15:02.590864 31501 raft_consensus.cc:2229] T 5d6936fc47ab4eada2b4311bc0b8257f P 68a3c2b17ed94e06aba66ed4178132ab [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:15:02.591885 31501 raft_consensus.cc:2258] T 5d6936fc47ab4eada2b4311bc0b8257f P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:02.593757 31501 ts_tablet_manager.cc:1875] T 5d6936fc47ab4eada2b4311bc0b8257f P 68a3c2b17ed94e06aba66ed4178132ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:02.598672 31502 tablet_replica.cc:330] T 5d6936fc47ab4eada2b4311bc0b8257f P 4092902b3e674857a0147f1b4f2cdb86: stopping tablet replica
I20240502 14:15:02.599560 31502 raft_consensus.cc:2229] T 5d6936fc47ab4eada2b4311bc0b8257f P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:02.610406 31502 raft_consensus.cc:2258] T 5d6936fc47ab4eada2b4311bc0b8257f P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:02.611670 31502 ts_tablet_manager.cc:1875] T 5d6936fc47ab4eada2b4311bc0b8257f P 4092902b3e674857a0147f1b4f2cdb86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:02.618993 31502 ts_tablet_manager.cc:1888] T 5d6936fc47ab4eada2b4311bc0b8257f P 4092902b3e674857a0147f1b4f2cdb86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:02.620915 31500 tablet_replica.cc:330] T 5d6936fc47ab4eada2b4311bc0b8257f P 1cb67158133e4648a6eceba32d4dbba3: stopping tablet replica
I20240502 14:15:02.621238 31500 raft_consensus.cc:2229] T 5d6936fc47ab4eada2b4311bc0b8257f P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:02.621582 31500 raft_consensus.cc:2258] T 5d6936fc47ab4eada2b4311bc0b8257f P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:02.622601 31500 ts_tablet_manager.cc:1875] T 5d6936fc47ab4eada2b4311bc0b8257f P 1cb67158133e4648a6eceba32d4dbba3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:02.623777 31501 ts_tablet_manager.cc:1888] T 5d6936fc47ab4eada2b4311bc0b8257f P 68a3c2b17ed94e06aba66ed4178132ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:02.623965 31501 log.cc:1196] T 5d6936fc47ab4eada2b4311bc0b8257f P 68a3c2b17ed94e06aba66ed4178132ab: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal/wals/5d6936fc47ab4eada2b4311bc0b8257f
I20240502 14:15:02.624462 31501 ts_tablet_manager.cc:1909] T 5d6936fc47ab4eada2b4311bc0b8257f P 68a3c2b17ed94e06aba66ed4178132ab: Deleting consensus metadata
I20240502 14:15:02.625722 26272 catalog_manager.cc:4830] TS 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): tablet 5d6936fc47ab4eada2b4311bc0b8257f (table rebalancer_dynamic_table_new_name [id=98bc01f98f5f427e893dd681fe235c24]) successfully deleted
I20240502 14:15:02.626318 31502 log.cc:1196] T 5d6936fc47ab4eada2b4311bc0b8257f P 4092902b3e674857a0147f1b4f2cdb86: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/wal/wals/5d6936fc47ab4eada2b4311bc0b8257f
I20240502 14:15:02.627133 31502 ts_tablet_manager.cc:1909] T 5d6936fc47ab4eada2b4311bc0b8257f P 4092902b3e674857a0147f1b4f2cdb86: Deleting consensus metadata
I20240502 14:15:02.630738 26272 catalog_manager.cc:4830] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097): tablet 5d6936fc47ab4eada2b4311bc0b8257f (table rebalancer_dynamic_table_new_name [id=98bc01f98f5f427e893dd681fe235c24]) successfully deleted
I20240502 14:15:02.658896 31500 ts_tablet_manager.cc:1888] T 5d6936fc47ab4eada2b4311bc0b8257f P 1cb67158133e4648a6eceba32d4dbba3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:02.659291 31500 log.cc:1196] T 5d6936fc47ab4eada2b4311bc0b8257f P 1cb67158133e4648a6eceba32d4dbba3: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal/wals/5d6936fc47ab4eada2b4311bc0b8257f
I20240502 14:15:02.659965 31500 ts_tablet_manager.cc:1909] T 5d6936fc47ab4eada2b4311bc0b8257f P 1cb67158133e4648a6eceba32d4dbba3: Deleting consensus metadata
I20240502 14:15:02.661798 26271 catalog_manager.cc:4830] TS 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): tablet 5d6936fc47ab4eada2b4311bc0b8257f (table rebalancer_dynamic_table_new_name [id=98bc01f98f5f427e893dd681fe235c24]) successfully deleted
I20240502 14:15:02.689975 26285 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:15:02.700769 29553 tablet_service.cc:1467] Processing CreateTablet for tablet c1f97e97e3c04a118240479e9bb1e212 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1072ed01b91245f29f6b097fdff5a0e9]), partition=
I20240502 14:15:02.700982 29423 tablet_service.cc:1467] Processing CreateTablet for tablet c1f97e97e3c04a118240479e9bb1e212 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1072ed01b91245f29f6b097fdff5a0e9]), partition=
I20240502 14:15:02.701221 29553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1f97e97e3c04a118240479e9bb1e212. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:02.701333 29423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1f97e97e3c04a118240479e9bb1e212. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:02.704475 29156 tablet_service.cc:1467] Processing CreateTablet for tablet c1f97e97e3c04a118240479e9bb1e212 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1072ed01b91245f29f6b097fdff5a0e9]), partition=
I20240502 14:15:02.705217 29156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1f97e97e3c04a118240479e9bb1e212. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:02.708995 31515 tablet_bootstrap.cc:492] T c1f97e97e3c04a118240479e9bb1e212 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:15:02.711516 31515 tablet_bootstrap.cc:654] T c1f97e97e3c04a118240479e9bb1e212 P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:02.723093 31302 tablet_bootstrap.cc:492] T c1f97e97e3c04a118240479e9bb1e212 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:15:02.724866 31302 tablet_bootstrap.cc:654] T c1f97e97e3c04a118240479e9bb1e212 P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:02.727424 31517 tablet_bootstrap.cc:492] T c1f97e97e3c04a118240479e9bb1e212 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:15:02.734659 31517 tablet_bootstrap.cc:654] T c1f97e97e3c04a118240479e9bb1e212 P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:02.737394 31479 consensus_queue.cc:1035] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:02.744812 31203 consensus_queue.cc:1035] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:02.752871 31517 tablet_bootstrap.cc:492] T c1f97e97e3c04a118240479e9bb1e212 P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:15:02.753067 31517 ts_tablet_manager.cc:1367] T c1f97e97e3c04a118240479e9bb1e212 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.008s
I20240502 14:15:02.754065 31517 raft_consensus.cc:348] T c1f97e97e3c04a118240479e9bb1e212 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:02.754280 31517 raft_consensus.cc:374] T c1f97e97e3c04a118240479e9bb1e212 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:02.754377 31517 raft_consensus.cc:729] T c1f97e97e3c04a118240479e9bb1e212 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:15:02.754699 31517 consensus_queue.cc:260] T c1f97e97e3c04a118240479e9bb1e212 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:02.755271 31517 ts_tablet_manager.cc:1398] T c1f97e97e3c04a118240479e9bb1e212 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240502 14:15:02.769908 31515 tablet_bootstrap.cc:492] T c1f97e97e3c04a118240479e9bb1e212 P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:15:02.770249 31515 ts_tablet_manager.cc:1367] T c1f97e97e3c04a118240479e9bb1e212 P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.061s user 0.001s sys 0.009s
I20240502 14:15:02.771165 31302 tablet_bootstrap.cc:492] T c1f97e97e3c04a118240479e9bb1e212 P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:15:02.771333 31302 ts_tablet_manager.cc:1367] T c1f97e97e3c04a118240479e9bb1e212 P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.048s user 0.009s sys 0.000s
I20240502 14:15:02.771363 31515 raft_consensus.cc:348] T c1f97e97e3c04a118240479e9bb1e212 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:02.771606 31515 raft_consensus.cc:374] T c1f97e97e3c04a118240479e9bb1e212 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:02.771816 31515 raft_consensus.cc:729] T c1f97e97e3c04a118240479e9bb1e212 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:15:02.772248 31302 raft_consensus.cc:348] T c1f97e97e3c04a118240479e9bb1e212 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:02.772504 31302 raft_consensus.cc:374] T c1f97e97e3c04a118240479e9bb1e212 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:02.772624 31302 raft_consensus.cc:729] T c1f97e97e3c04a118240479e9bb1e212 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:15:02.772859 31302 consensus_queue.cc:260] T c1f97e97e3c04a118240479e9bb1e212 P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:02.773365 31302 ts_tablet_manager.cc:1398] T c1f97e97e3c04a118240479e9bb1e212 P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:02.775017 31515 consensus_queue.cc:260] T c1f97e97e3c04a118240479e9bb1e212 P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:02.781106 31515 ts_tablet_manager.cc:1398] T c1f97e97e3c04a118240479e9bb1e212 P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240502 14:15:02.784703 26817 consensus_queue.cc:237] T 981e25d29abc44299dc115d6df835c58 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:02.789171 29176 raft_consensus.cc:1261] T 981e25d29abc44299dc115d6df835c58 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:02.789870 31192 consensus_queue.cc:1035] T 981e25d29abc44299dc115d6df835c58 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:02.790289 26423 raft_consensus.cc:1261] T 981e25d29abc44299dc115d6df835c58 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:02.790838 31192 consensus_queue.cc:1035] T 981e25d29abc44299dc115d6df835c58 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:02.793154 31357 raft_consensus.cc:2932] T 981e25d29abc44299dc115d6df835c58 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:02.794011 29176 raft_consensus.cc:2932] T 981e25d29abc44299dc115d6df835c58 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:02.801798 26285 catalog_manager.cc:5484] T 981e25d29abc44299dc115d6df835c58 P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: config changed from index 4 to 5, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:02.805061 26423 raft_consensus.cc:2932] T 981e25d29abc44299dc115d6df835c58 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:02.808943 26668 tablet_service.cc:1514] Processing DeleteTablet for tablet 981e25d29abc44299dc115d6df835c58 with delete_type TABLET_DATA_TOMBSTONED (TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34150
I20240502 14:15:02.815554 31532 tablet_replica.cc:330] T 981e25d29abc44299dc115d6df835c58 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: stopping tablet replica
I20240502 14:15:02.815834 31532 raft_consensus.cc:2229] T 981e25d29abc44299dc115d6df835c58 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:02.816190 31532 raft_consensus.cc:2258] T 981e25d29abc44299dc115d6df835c58 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:02.817008 31532 ts_tablet_manager.cc:1875] T 981e25d29abc44299dc115d6df835c58 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:02.822918 31532 ts_tablet_manager.cc:1888] T 981e25d29abc44299dc115d6df835c58 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:15:02.823068 31532 log.cc:1196] T 981e25d29abc44299dc115d6df835c58 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/wals/981e25d29abc44299dc115d6df835c58
I20240502 14:15:02.823868 26272 catalog_manager.cc:4830] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): tablet 981e25d29abc44299dc115d6df835c58 (table rebalance_test_table_19 [id=d4a5cfed144a42db8322f6823ef341a8]) successfully deleted
I20240502 14:15:02.972903 31217 consensus_queue.cc:1035] T 13136deb9046440f88bf3471ff42312c P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:02.978607 31217 consensus_queue.cc:1035] T 13136deb9046440f88bf3471ff42312c P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:03.005941 30953 consensus_queue.cc:1035] T f6561a18519245f9a99690259c9c295e P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:03.013325 30953 consensus_queue.cc:1035] T f6561a18519245f9a99690259c9c295e P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:03.122675 26279 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_extra_table_28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:03.134011 29553 tablet_service.cc:1467] Processing CreateTablet for tablet b8430c13474b4850b71020c4a89bea68 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=0dc6f24955544368ae080f1a95f9589b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:03.134578 29553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8430c13474b4850b71020c4a89bea68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:03.136893 26537 tablet_service.cc:1467] Processing CreateTablet for tablet b8430c13474b4850b71020c4a89bea68 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=0dc6f24955544368ae080f1a95f9589b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:03.137265 26537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8430c13474b4850b71020c4a89bea68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:03.138480 26407 tablet_service.cc:1467] Processing CreateTablet for tablet a04b8c49626e491b845d9c81142ac659 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=0dc6f24955544368ae080f1a95f9589b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:03.138988 26407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a04b8c49626e491b845d9c81142ac659. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:03.141260 29156 tablet_service.cc:1467] Processing CreateTablet for tablet 0c1db9ae649f4ac0bbe05fd07eba7df6 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=0dc6f24955544368ae080f1a95f9589b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:03.141804 29156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c1db9ae649f4ac0bbe05fd07eba7df6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:03.144767 29423 tablet_service.cc:1467] Processing CreateTablet for tablet b8430c13474b4850b71020c4a89bea68 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=0dc6f24955544368ae080f1a95f9589b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:03.145233 29423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8430c13474b4850b71020c4a89bea68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:03.151067 31302 tablet_bootstrap.cc:492] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:15:03.151324 29554 tablet_service.cc:1467] Processing CreateTablet for tablet 0c1db9ae649f4ac0bbe05fd07eba7df6 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=0dc6f24955544368ae080f1a95f9589b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:03.151829 29554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c1db9ae649f4ac0bbe05fd07eba7df6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:03.152976 31302 tablet_bootstrap.cc:654] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:03.153923 31582 tablet_bootstrap.cc:492] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:15:03.156201 31582 tablet_bootstrap.cc:654] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:03.157472 29421 tablet_service.cc:1467] Processing CreateTablet for tablet 0c1db9ae649f4ac0bbe05fd07eba7df6 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=0dc6f24955544368ae080f1a95f9589b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:03.157971 29422 tablet_service.cc:1467] Processing CreateTablet for tablet a04b8c49626e491b845d9c81142ac659 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=0dc6f24955544368ae080f1a95f9589b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:03.157991 29421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c1db9ae649f4ac0bbe05fd07eba7df6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:03.163909 29422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a04b8c49626e491b845d9c81142ac659. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:03.166639 31585 tablet_bootstrap.cc:492] T a04b8c49626e491b845d9c81142ac659 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:15:03.168852 31585 tablet_bootstrap.cc:654] T a04b8c49626e491b845d9c81142ac659 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:03.171136 31302 tablet_bootstrap.cc:492] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:15:03.171366 31302 ts_tablet_manager.cc:1367] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.005s
I20240502 14:15:03.172183 31302 raft_consensus.cc:348] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:03.172457 31302 raft_consensus.cc:374] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:03.172541 31302 raft_consensus.cc:729] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:15:03.174170 31302 consensus_queue.cc:260] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:03.174782 31302 ts_tablet_manager.cc:1398] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:15:03.175452 26797 tablet_service.cc:1467] Processing CreateTablet for tablet a04b8c49626e491b845d9c81142ac659 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=0dc6f24955544368ae080f1a95f9589b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:03.176072 26797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a04b8c49626e491b845d9c81142ac659. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:03.178649 31585 tablet_bootstrap.cc:492] T a04b8c49626e491b845d9c81142ac659 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:15:03.178874 31585 ts_tablet_manager.cc:1367] T a04b8c49626e491b845d9c81142ac659 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20240502 14:15:03.179854 31585 raft_consensus.cc:348] T a04b8c49626e491b845d9c81142ac659 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:03.181604 31585 raft_consensus.cc:374] T a04b8c49626e491b845d9c81142ac659 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:03.181741 31585 raft_consensus.cc:729] T a04b8c49626e491b845d9c81142ac659 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:15:03.182008 31585 consensus_queue.cc:260] T a04b8c49626e491b845d9c81142ac659 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:03.182827 31585 ts_tablet_manager.cc:1398] T a04b8c49626e491b845d9c81142ac659 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:15:03.186559 31515 tablet_bootstrap.cc:492] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:15:03.195745 31587 tablet_bootstrap.cc:492] T a04b8c49626e491b845d9c81142ac659 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:15:03.196748 31517 tablet_bootstrap.cc:492] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:15:03.197487 31515 tablet_bootstrap.cc:654] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:03.198606 31517 tablet_bootstrap.cc:654] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:03.201668 31517 tablet_bootstrap.cc:492] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:15:03.201917 31517 ts_tablet_manager.cc:1367] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240502 14:15:03.202826 31517 raft_consensus.cc:348] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:03.203079 31517 raft_consensus.cc:374] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:03.203177 31517 raft_consensus.cc:729] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:15:03.203461 31517 consensus_queue.cc:260] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:03.204280 31515 tablet_bootstrap.cc:492] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:15:03.204550 31515 ts_tablet_manager.cc:1367] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20240502 14:15:03.204758 31582 tablet_bootstrap.cc:492] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:15:03.204937 31582 ts_tablet_manager.cc:1367] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.051s user 0.012s sys 0.027s
I20240502 14:15:03.205574 31517 ts_tablet_manager.cc:1398] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20240502 14:15:03.205590 31587 tablet_bootstrap.cc:654] T a04b8c49626e491b845d9c81142ac659 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:03.205618 31515 raft_consensus.cc:348] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:03.205855 31515 raft_consensus.cc:374] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:03.205948 31515 raft_consensus.cc:729] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:15:03.206192 31517 tablet_bootstrap.cc:492] T a04b8c49626e491b845d9c81142ac659 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:15:03.206177 31582 raft_consensus.cc:348] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:03.206393 31582 raft_consensus.cc:374] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:03.206635 31582 raft_consensus.cc:729] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:15:03.206920 31582 consensus_queue.cc:260] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:03.207602 31582 ts_tablet_manager.cc:1398] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:15:03.208931 31515 consensus_queue.cc:260] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:03.211347 31517 tablet_bootstrap.cc:654] T a04b8c49626e491b845d9c81142ac659 P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:03.213258 31515 ts_tablet_manager.cc:1398] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 14:15:03.213637 31515 tablet_bootstrap.cc:492] T b8430c13474b4850b71020c4a89bea68 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:15:03.215551 31515 tablet_bootstrap.cc:654] T b8430c13474b4850b71020c4a89bea68 P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:03.218892 31515 tablet_bootstrap.cc:492] T b8430c13474b4850b71020c4a89bea68 P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:15:03.219166 31515 ts_tablet_manager.cc:1367] T b8430c13474b4850b71020c4a89bea68 P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 14:15:03.219767 31517 tablet_bootstrap.cc:492] T a04b8c49626e491b845d9c81142ac659 P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:15:03.219957 31517 ts_tablet_manager.cc:1367] T a04b8c49626e491b845d9c81142ac659 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20240502 14:15:03.220216 31515 raft_consensus.cc:348] T b8430c13474b4850b71020c4a89bea68 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:03.220551 31515 raft_consensus.cc:374] T b8430c13474b4850b71020c4a89bea68 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:03.220747 31515 raft_consensus.cc:729] T b8430c13474b4850b71020c4a89bea68 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:15:03.220906 31517 raft_consensus.cc:348] T a04b8c49626e491b845d9c81142ac659 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:03.222627 31515 consensus_queue.cc:260] T b8430c13474b4850b71020c4a89bea68 P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:03.223441 31515 ts_tablet_manager.cc:1398] T b8430c13474b4850b71020c4a89bea68 P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:15:03.224510 31517 raft_consensus.cc:374] T a04b8c49626e491b845d9c81142ac659 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:03.224697 31517 raft_consensus.cc:729] T a04b8c49626e491b845d9c81142ac659 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:15:03.225137 31517 consensus_queue.cc:260] T a04b8c49626e491b845d9c81142ac659 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:03.225920 31517 ts_tablet_manager.cc:1398] T a04b8c49626e491b845d9c81142ac659 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:15:03.226302 31517 tablet_bootstrap.cc:492] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:15:03.228133 31517 tablet_bootstrap.cc:654] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:03.229346 31587 tablet_bootstrap.cc:492] T a04b8c49626e491b845d9c81142ac659 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:15:03.229588 31587 ts_tablet_manager.cc:1367] T a04b8c49626e491b845d9c81142ac659 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.000s
I20240502 14:15:03.230541 31587 raft_consensus.cc:348] T a04b8c49626e491b845d9c81142ac659 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:03.232199 31587 raft_consensus.cc:374] T a04b8c49626e491b845d9c81142ac659 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:03.232385 31587 raft_consensus.cc:729] T a04b8c49626e491b845d9c81142ac659 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:15:03.232805 31587 consensus_queue.cc:260] T a04b8c49626e491b845d9c81142ac659 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:03.234139 31419 raft_consensus.cc:482] T c1f97e97e3c04a118240479e9bb1e212 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:03.234331 31419 raft_consensus.cc:504] T c1f97e97e3c04a118240479e9bb1e212 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:03.235020 31419 leader_election.cc:288] T c1f97e97e3c04a118240479e9bb1e212 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:15:03.235606 29574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f97e97e3c04a118240479e9bb1e212" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86" is_pre_election: true
I20240502 14:15:03.235852 29574 raft_consensus.cc:2454] T c1f97e97e3c04a118240479e9bb1e212 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 0.
I20240502 14:15:03.236006 31587 ts_tablet_manager.cc:1398] T a04b8c49626e491b845d9c81142ac659 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:15:03.236217 29174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f97e97e3c04a118240479e9bb1e212" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:15:03.236477 29174 raft_consensus.cc:2454] T c1f97e97e3c04a118240479e9bb1e212 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 0.
I20240502 14:15:03.236976 29378 leader_election.cc:302] T c1f97e97e3c04a118240479e9bb1e212 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 68a3c2b17ed94e06aba66ed4178132ab; no voters:
I20240502 14:15:03.237731 31419 raft_consensus.cc:2781] T c1f97e97e3c04a118240479e9bb1e212 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:03.237847 31419 raft_consensus.cc:482] T c1f97e97e3c04a118240479e9bb1e212 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:03.237952 31419 raft_consensus.cc:3037] T c1f97e97e3c04a118240479e9bb1e212 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:03.248576 31517 tablet_bootstrap.cc:492] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:15:03.248795 31517 ts_tablet_manager.cc:1367] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.000s
I20240502 14:15:03.249755 31517 raft_consensus.cc:348] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:03.250028 31517 raft_consensus.cc:374] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:03.250104 31517 raft_consensus.cc:729] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:15:03.250375 31517 consensus_queue.cc:260] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:03.252741 31419 raft_consensus.cc:504] T c1f97e97e3c04a118240479e9bb1e212 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:03.255404 31517 ts_tablet_manager.cc:1398] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 14:15:03.256670 29174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f97e97e3c04a118240479e9bb1e212" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:15:03.256927 29174 raft_consensus.cc:3037] T c1f97e97e3c04a118240479e9bb1e212 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:03.259409 29174 raft_consensus.cc:2454] T c1f97e97e3c04a118240479e9bb1e212 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 1.
I20240502 14:15:03.259927 29378 leader_election.cc:302] T c1f97e97e3c04a118240479e9bb1e212 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 68a3c2b17ed94e06aba66ed4178132ab; no voters:
I20240502 14:15:03.260568 31595 raft_consensus.cc:2781] T c1f97e97e3c04a118240479e9bb1e212 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:03.279480 29574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f97e97e3c04a118240479e9bb1e212" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86"
I20240502 14:15:03.279762 29574 raft_consensus.cc:3037] T c1f97e97e3c04a118240479e9bb1e212 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:03.283146 29574 raft_consensus.cc:2454] T c1f97e97e3c04a118240479e9bb1e212 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 1.
I20240502 14:15:03.284723 31419 leader_election.cc:288] T c1f97e97e3c04a118240479e9bb1e212 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 election: Requested vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:15:03.312590 31595 raft_consensus.cc:686] T c1f97e97e3c04a118240479e9bb1e212 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 LEADER]: Becoming Leader. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Running, Role: LEADER
I20240502 14:15:03.312983 31595 consensus_queue.cc:237] T c1f97e97e3c04a118240479e9bb1e212 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:03.319000 26285 catalog_manager.cc:5484] T c1f97e97e3c04a118240479e9bb1e212 P 68a3c2b17ed94e06aba66ed4178132ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199). New cstate: current_term: 1 leader_uuid: "68a3c2b17ed94e06aba66ed4178132ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:15:03.377805 29156 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:03.382018 31203 raft_consensus.cc:482] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:03.382292 31203 raft_consensus.cc:504] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:03.383074 31203 leader_election.cc:288] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:15:03.385052 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c1db9ae649f4ac0bbe05fd07eba7df6" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:15:03.385500 29443 raft_consensus.cc:2454] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:15:03.386819 29174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c1db9ae649f4ac0bbe05fd07eba7df6" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:15:03.386876 29511 leader_election.cc:302] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, 68a3c2b17ed94e06aba66ed4178132ab; no voters:
I20240502 14:15:03.387058 29174 raft_consensus.cc:2454] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:15:03.388144 31203 raft_consensus.cc:2781] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:03.388279 31203 raft_consensus.cc:482] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:03.388442 31203 raft_consensus.cc:3037] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:03.391757 29553 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:03.397670 31203 raft_consensus.cc:504] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:03.401777 31203 leader_election.cc:288] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Requested vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:15:03.402164 29174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c1db9ae649f4ac0bbe05fd07eba7df6" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:15:03.402462 29174 raft_consensus.cc:3037] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:03.404188 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c1db9ae649f4ac0bbe05fd07eba7df6" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
I20240502 14:15:03.404469 29443 raft_consensus.cc:3037] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:03.406760 26668 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:15:03.407748 29443 raft_consensus.cc:2454] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:15:03.408246 29511 leader_election.cc:302] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, 68a3c2b17ed94e06aba66ed4178132ab; no voters:
I20240502 14:15:03.408668 29174 raft_consensus.cc:2454] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:15:03.410812 31203 raft_consensus.cc:2781] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:03.411140 31203 raft_consensus.cc:686] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Becoming Leader. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Running, Role: LEADER
I20240502 14:15:03.411509 31203 consensus_queue.cc:237] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:03.443457 26407 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:03.445691 26537 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:03.448412 29421 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:15:03.452535 31595 raft_consensus.cc:482] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:03.452827 31595 raft_consensus.cc:504] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:03.453852 31595 leader_election.cc:288] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:15:03.454792 29574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8430c13474b4850b71020c4a89bea68" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86" is_pre_election: true
I20240502 14:15:03.455142 29574 raft_consensus.cc:2454] T b8430c13474b4850b71020c4a89bea68 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 0.
I20240502 14:15:03.455761 26557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8430c13474b4850b71020c4a89bea68" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:15:03.456039 26557 raft_consensus.cc:2454] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 0.
I20240502 14:15:03.456851 29378 leader_election.cc:302] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:15:03.458590 31595 raft_consensus.cc:2781] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:03.458736 31595 raft_consensus.cc:482] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:03.458873 31595 raft_consensus.cc:3037] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:03.471411 31595 raft_consensus.cc:504] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:03.472971 29574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8430c13474b4850b71020c4a89bea68" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86"
I20240502 14:15:03.473259 29574 raft_consensus.cc:3037] T b8430c13474b4850b71020c4a89bea68 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:03.482297 29574 raft_consensus.cc:2454] T b8430c13474b4850b71020c4a89bea68 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 1.
I20240502 14:15:03.483322 29379 leader_election.cc:302] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, 68a3c2b17ed94e06aba66ed4178132ab; no voters:
I20240502 14:15:03.483836 31596 raft_consensus.cc:2781] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:03.484000 31596 raft_consensus.cc:686] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 LEADER]: Becoming Leader. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Running, Role: LEADER
I20240502 14:15:03.484423 31596 consensus_queue.cc:237] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:03.495929 31173 raft_consensus.cc:482] T a04b8c49626e491b845d9c81142ac659 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:03.496152 31173 raft_consensus.cc:504] T a04b8c49626e491b845d9c81142ac659 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:03.497547 31173 leader_election.cc:288] T a04b8c49626e491b845d9c81142ac659 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:15:03.498185 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a04b8c49626e491b845d9c81142ac659" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:15:03.498526 29443 raft_consensus.cc:2454] T a04b8c49626e491b845d9c81142ac659 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:15:03.501705 26363 leader_election.cc:302] T a04b8c49626e491b845d9c81142ac659 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:15:03.502694 26819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a04b8c49626e491b845d9c81142ac659" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:15:03.502933 26819 raft_consensus.cc:2454] T a04b8c49626e491b845d9c81142ac659 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:15:03.504850 31499 raft_consensus.cc:2781] T a04b8c49626e491b845d9c81142ac659 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:03.504983 31499 raft_consensus.cc:482] T a04b8c49626e491b845d9c81142ac659 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:03.505120 31499 raft_consensus.cc:3037] T a04b8c49626e491b845d9c81142ac659 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:03.508854 31499 raft_consensus.cc:504] T a04b8c49626e491b845d9c81142ac659 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:03.509763 31499 leader_election.cc:288] T a04b8c49626e491b845d9c81142ac659 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Requested vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:15:03.510828 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a04b8c49626e491b845d9c81142ac659" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
I20240502 14:15:03.511140 29443 raft_consensus.cc:3037] T a04b8c49626e491b845d9c81142ac659 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:03.515089 26819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a04b8c49626e491b845d9c81142ac659" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:15:03.515327 26819 raft_consensus.cc:3037] T a04b8c49626e491b845d9c81142ac659 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:03.515935 29443 raft_consensus.cc:2454] T a04b8c49626e491b845d9c81142ac659 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:15:03.516539 26363 leader_election.cc:302] T a04b8c49626e491b845d9c81142ac659 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:15:03.516889 31499 raft_consensus.cc:2781] T a04b8c49626e491b845d9c81142ac659 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:03.517081 31499 raft_consensus.cc:686] T a04b8c49626e491b845d9c81142ac659 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Becoming Leader. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Running, Role: LEADER
I20240502 14:15:03.517374 31499 consensus_queue.cc:237] T a04b8c49626e491b845d9c81142ac659 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:03.518625 26819 raft_consensus.cc:2454] T a04b8c49626e491b845d9c81142ac659 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:15:03.526501 31595 leader_election.cc:288] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 election: Requested vote from peers 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:15:03.527554 26557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8430c13474b4850b71020c4a89bea68" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:15:03.527880 26557 raft_consensus.cc:3037] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:03.536582 26797 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:03.537132 26285 catalog_manager.cc:5484] T a04b8c49626e491b845d9c81142ac659 P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: term changed from 0 to 1, leader changed from <none> to 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193). New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:03.546270 26285 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:15:03.548743 26557 raft_consensus.cc:2454] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 1.
I20240502 14:15:03.599355 29289 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:15:03.599788 29176 raft_consensus.cc:1261] T c1f97e97e3c04a118240479e9bb1e212 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:03.600487 31595 consensus_queue.cc:1035] T c1f97e97e3c04a118240479e9bb1e212 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:03.616212 29574 raft_consensus.cc:1261] T c1f97e97e3c04a118240479e9bb1e212 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:15:03.619571 26362 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.194:36025, user_credentials={real_user=slave}} blocked reactor thread for 100721us
W20240502 14:15:03.620317 29510 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.197:38715, user_credentials={real_user=slave}} blocked reactor thread for 190671us
I20240502 14:15:03.620323 26285 catalog_manager.cc:5484] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199). New cstate: current_term: 1 leader_uuid: "68a3c2b17ed94e06aba66ed4178132ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:15:03.635589 31596 consensus_queue.cc:1035] T c1f97e97e3c04a118240479e9bb1e212 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:03.704869 26285 catalog_manager.cc:5484] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200). New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:15:03.715741 26285 catalog_manager.cc:5576] TS 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715) does not have the latest schema for tablet c1f97e97e3c04a118240479e9bb1e212 (table rebalancer_dynamic_table [id=1072ed01b91245f29f6b097fdff5a0e9]). Expected version 1 got 0
I20240502 14:15:03.726805 31636 tablet.cc:1722] T c1f97e97e3c04a118240479e9bb1e212 P 68a3c2b17ed94e06aba66ed4178132ab: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:03.727874 31636 tablet.cc:1620] T c1f97e97e3c04a118240479e9bb1e212 P 68a3c2b17ed94e06aba66ed4178132ab: MemRowSet was empty: no flush needed.
I20240502 14:15:03.735975 26279 catalog_manager.cc:5576] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097) does not have the latest schema for tablet c1f97e97e3c04a118240479e9bb1e212 (table rebalancer_dynamic_table [id=1072ed01b91245f29f6b097fdff5a0e9]). Expected version 1 got 0
I20240502 14:15:03.739240 26285 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=1072ed01b91245f29f6b097fdff5a0e9] alter complete (version 1)
I20240502 14:15:03.755450 31638 tablet.cc:1722] T c1f97e97e3c04a118240479e9bb1e212 P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:03.756075 31638 tablet.cc:1620] T c1f97e97e3c04a118240479e9bb1e212 P 4092902b3e674857a0147f1b4f2cdb86: MemRowSet was empty: no flush needed.
I20240502 14:15:03.758363 26279 catalog_manager.cc:5576] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097) does not have the latest schema for tablet c1f97e97e3c04a118240479e9bb1e212 (table rebalancer_dynamic_table [id=1072ed01b91245f29f6b097fdff5a0e9]). Expected version 1 got 0
W20240502 14:15:03.762358 29244 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.193:37985, user_credentials={real_user=slave}} blocked reactor thread for 67574.5us
I20240502 14:15:03.764940 31637 tablet.cc:1722] T c1f97e97e3c04a118240479e9bb1e212 P 1cb67158133e4648a6eceba32d4dbba3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:03.765601 31637 tablet.cc:1620] T c1f97e97e3c04a118240479e9bb1e212 P 1cb67158133e4648a6eceba32d4dbba3: MemRowSet was empty: no flush needed.
I20240502 14:15:03.770355 26279 catalog_manager.cc:5576] TS 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715) does not have the latest schema for tablet c1f97e97e3c04a118240479e9bb1e212 (table rebalancer_dynamic_table [id=1072ed01b91245f29f6b097fdff5a0e9]). Expected version 1 got 0
I20240502 14:15:03.846850 26276 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:15:03.848407 26427 consensus_queue.cc:237] T 226b35583ee4441a82a66520aeffbb7d P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:03.851356 29574 raft_consensus.cc:1261] T 226b35583ee4441a82a66520aeffbb7d P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:03.851328 26689 raft_consensus.cc:1261] T 226b35583ee4441a82a66520aeffbb7d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:03.852043 31173 consensus_queue.cc:1035] T 226b35583ee4441a82a66520aeffbb7d P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:03.853729 31173 consensus_queue.cc:1035] T 226b35583ee4441a82a66520aeffbb7d P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:03.856719 31173 raft_consensus.cc:2932] T 226b35583ee4441a82a66520aeffbb7d P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:15:03.860101 31636 tablet.cc:1722] T c1f97e97e3c04a118240479e9bb1e212 P 68a3c2b17ed94e06aba66ed4178132ab: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:03.861132 31636 tablet.cc:1620] T c1f97e97e3c04a118240479e9bb1e212 P 68a3c2b17ed94e06aba66ed4178132ab: MemRowSet was empty: no flush needed.
I20240502 14:15:03.861604 26688 raft_consensus.cc:2932] T 226b35583ee4441a82a66520aeffbb7d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:15:03.862057 29574 raft_consensus.cc:2932] T 226b35583ee4441a82a66520aeffbb7d P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:15:03.864624 31596 consensus_queue.cc:1035] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:03.866503 31638 tablet.cc:1722] T c1f97e97e3c04a118240479e9bb1e212 P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:03.867050 31638 tablet.cc:1620] T c1f97e97e3c04a118240479e9bb1e212 P 4092902b3e674857a0147f1b4f2cdb86: MemRowSet was empty: no flush needed.
I20240502 14:15:03.872241 26285 catalog_manager.cc:5484] T 226b35583ee4441a82a66520aeffbb7d P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: config changed from index 4 to 5, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) evicted. New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:15:03.878785 26537 tablet_service.cc:1514] Processing DeleteTablet for tablet 226b35583ee4441a82a66520aeffbb7d with delete_type TABLET_DATA_TOMBSTONED (TS ba187e38a2754d60a12de9ac4856430a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51696
I20240502 14:15:03.879837 31661 tablet_replica.cc:330] T 226b35583ee4441a82a66520aeffbb7d P ba187e38a2754d60a12de9ac4856430a: stopping tablet replica
I20240502 14:15:03.880090 31661 raft_consensus.cc:2229] T 226b35583ee4441a82a66520aeffbb7d P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:03.880474 31661 raft_consensus.cc:2258] T 226b35583ee4441a82a66520aeffbb7d P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:03.881317 31661 ts_tablet_manager.cc:1875] T 226b35583ee4441a82a66520aeffbb7d P ba187e38a2754d60a12de9ac4856430a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:03.886664 31637 tablet.cc:1722] T c1f97e97e3c04a118240479e9bb1e212 P 1cb67158133e4648a6eceba32d4dbba3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:03.887240 31637 tablet.cc:1620] T c1f97e97e3c04a118240479e9bb1e212 P 1cb67158133e4648a6eceba32d4dbba3: MemRowSet was empty: no flush needed.
I20240502 14:15:03.889999 31661 ts_tablet_manager.cc:1888] T 226b35583ee4441a82a66520aeffbb7d P ba187e38a2754d60a12de9ac4856430a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:15:03.890249 31661 log.cc:1196] T 226b35583ee4441a82a66520aeffbb7d P ba187e38a2754d60a12de9ac4856430a: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/226b35583ee4441a82a66520aeffbb7d
I20240502 14:15:03.891301 26271 catalog_manager.cc:4830] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): tablet 226b35583ee4441a82a66520aeffbb7d (table rebalance_test_table_5 [id=8069f2786bf84bf4a012c26e82757ad4]) successfully deleted
I20240502 14:15:03.891564 31596 consensus_queue.cc:1035] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:03.901082 26279 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=1072ed01b91245f29f6b097fdff5a0e9] alter complete (version 2)
I20240502 14:15:03.998967 31623 consensus_queue.cc:1035] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:04.007042 31620 consensus_queue.cc:1035] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:04.007156 26279 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:15:04.014189 31628 consensus_queue.cc:1035] T a04b8c49626e491b845d9c81142ac659 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:04.017556 31636 tablet.cc:1722] T c1f97e97e3c04a118240479e9bb1e212 P 68a3c2b17ed94e06aba66ed4178132ab: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:04.019501 31637 tablet.cc:1722] T c1f97e97e3c04a118240479e9bb1e212 P 1cb67158133e4648a6eceba32d4dbba3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:04.029942 31638 tablet.cc:1722] T c1f97e97e3c04a118240479e9bb1e212 P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:04.033504 31628 consensus_queue.cc:1035] T a04b8c49626e491b845d9c81142ac659 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:04.077271 26276 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table_new_name [id=1072ed01b91245f29f6b097fdff5a0e9] alter complete (version 3)
I20240502 14:15:04.182166 26279 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:04.182478 26279 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:04.185736 26279 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet c1f97e97e3c04a118240479e9bb1e212
I20240502 14:15:04.186609 29156 tablet_service.cc:1514] Processing DeleteTablet for tablet c1f97e97e3c04a118240479e9bb1e212 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:04 UTC) from {username='slave'} at 127.0.0.1:55202
I20240502 14:15:04.186887 29553 tablet_service.cc:1514] Processing DeleteTablet for tablet c1f97e97e3c04a118240479e9bb1e212 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:04 UTC) from {username='slave'} at 127.0.0.1:36398
I20240502 14:15:04.187275 29421 tablet_service.cc:1514] Processing DeleteTablet for tablet c1f97e97e3c04a118240479e9bb1e212 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:04 UTC) from {username='slave'} at 127.0.0.1:40652
I20240502 14:15:04.187319 31718 tablet_replica.cc:330] T c1f97e97e3c04a118240479e9bb1e212 P 1cb67158133e4648a6eceba32d4dbba3: stopping tablet replica
I20240502 14:15:04.187472 31719 tablet_replica.cc:330] T c1f97e97e3c04a118240479e9bb1e212 P 4092902b3e674857a0147f1b4f2cdb86: stopping tablet replica
I20240502 14:15:04.187621 31718 raft_consensus.cc:2229] T c1f97e97e3c04a118240479e9bb1e212 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:04.187752 31719 raft_consensus.cc:2229] T c1f97e97e3c04a118240479e9bb1e212 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:04.187911 31718 raft_consensus.cc:2258] T c1f97e97e3c04a118240479e9bb1e212 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:04.188023 31720 tablet_replica.cc:330] T c1f97e97e3c04a118240479e9bb1e212 P 68a3c2b17ed94e06aba66ed4178132ab: stopping tablet replica
I20240502 14:15:04.188086 31719 raft_consensus.cc:2258] T c1f97e97e3c04a118240479e9bb1e212 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:04.188370 31720 raft_consensus.cc:2229] T c1f97e97e3c04a118240479e9bb1e212 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:15:04.188871 31720 raft_consensus.cc:2258] T c1f97e97e3c04a118240479e9bb1e212 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:04.188961 31718 ts_tablet_manager.cc:1875] T c1f97e97e3c04a118240479e9bb1e212 P 1cb67158133e4648a6eceba32d4dbba3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:04.188961 31719 ts_tablet_manager.cc:1875] T c1f97e97e3c04a118240479e9bb1e212 P 4092902b3e674857a0147f1b4f2cdb86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:04.189795 31720 ts_tablet_manager.cc:1875] T c1f97e97e3c04a118240479e9bb1e212 P 68a3c2b17ed94e06aba66ed4178132ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:04.196409 31718 ts_tablet_manager.cc:1888] T c1f97e97e3c04a118240479e9bb1e212 P 1cb67158133e4648a6eceba32d4dbba3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:04.196621 31718 log.cc:1196] T c1f97e97e3c04a118240479e9bb1e212 P 1cb67158133e4648a6eceba32d4dbba3: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal/wals/c1f97e97e3c04a118240479e9bb1e212
I20240502 14:15:04.197160 31718 ts_tablet_manager.cc:1909] T c1f97e97e3c04a118240479e9bb1e212 P 1cb67158133e4648a6eceba32d4dbba3: Deleting consensus metadata
I20240502 14:15:04.198277 31720 ts_tablet_manager.cc:1888] T c1f97e97e3c04a118240479e9bb1e212 P 68a3c2b17ed94e06aba66ed4178132ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:04.198280 31719 ts_tablet_manager.cc:1888] T c1f97e97e3c04a118240479e9bb1e212 P 4092902b3e674857a0147f1b4f2cdb86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:04.198419 31719 log.cc:1196] T c1f97e97e3c04a118240479e9bb1e212 P 4092902b3e674857a0147f1b4f2cdb86: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/wal/wals/c1f97e97e3c04a118240479e9bb1e212
I20240502 14:15:04.198400 26271 catalog_manager.cc:4830] TS 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): tablet c1f97e97e3c04a118240479e9bb1e212 (table rebalancer_dynamic_table_new_name [id=1072ed01b91245f29f6b097fdff5a0e9]) successfully deleted
I20240502 14:15:04.198549 31720 log.cc:1196] T c1f97e97e3c04a118240479e9bb1e212 P 68a3c2b17ed94e06aba66ed4178132ab: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal/wals/c1f97e97e3c04a118240479e9bb1e212
I20240502 14:15:04.198954 31719 ts_tablet_manager.cc:1909] T c1f97e97e3c04a118240479e9bb1e212 P 4092902b3e674857a0147f1b4f2cdb86: Deleting consensus metadata
I20240502 14:15:04.199098 31720 ts_tablet_manager.cc:1909] T c1f97e97e3c04a118240479e9bb1e212 P 68a3c2b17ed94e06aba66ed4178132ab: Deleting consensus metadata
I20240502 14:15:04.199925 26272 catalog_manager.cc:4830] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097): tablet c1f97e97e3c04a118240479e9bb1e212 (table rebalancer_dynamic_table_new_name [id=1072ed01b91245f29f6b097fdff5a0e9]) successfully deleted
I20240502 14:15:04.200189 26272 catalog_manager.cc:4830] TS 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): tablet c1f97e97e3c04a118240479e9bb1e212 (table rebalancer_dynamic_table_new_name [id=1072ed01b91245f29f6b097fdff5a0e9]) successfully deleted
I20240502 14:15:04.290851 26279 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_extra_table_30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:04.291030 26277 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:15:04.320647 29156 tablet_service.cc:1467] Processing CreateTablet for tablet 5808efbc3ef84f3b862045c67b30cd8b (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=50fafe33ec534b21a087e7ea3eab5a91]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:04.321249 29156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5808efbc3ef84f3b862045c67b30cd8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:04.321321 29553 tablet_service.cc:1467] Processing CreateTablet for tablet 5808efbc3ef84f3b862045c67b30cd8b (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=50fafe33ec534b21a087e7ea3eab5a91]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:04.321698 29289 tablet_service.cc:1467] Processing CreateTablet for tablet 5808efbc3ef84f3b862045c67b30cd8b (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=50fafe33ec534b21a087e7ea3eab5a91]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:04.321837 29553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5808efbc3ef84f3b862045c67b30cd8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:04.322281 29289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5808efbc3ef84f3b862045c67b30cd8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:04.327970 29288 tablet_service.cc:1467] Processing CreateTablet for tablet 736ed8faceec4f0fa07c62952ea32bf6 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=50fafe33ec534b21a087e7ea3eab5a91]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:04.328518 29288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 736ed8faceec4f0fa07c62952ea32bf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:04.329906 31733 tablet_bootstrap.cc:492] T 5808efbc3ef84f3b862045c67b30cd8b P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:15:04.334347 29554 tablet_service.cc:1467] Processing CreateTablet for tablet 736ed8faceec4f0fa07c62952ea32bf6 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=50fafe33ec534b21a087e7ea3eab5a91]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:04.335000 29554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 736ed8faceec4f0fa07c62952ea32bf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:04.344146 31734 tablet_bootstrap.cc:492] T 5808efbc3ef84f3b862045c67b30cd8b P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:15:04.346513 29287 tablet_service.cc:1467] Processing CreateTablet for tablet 32b94809b6a841cca35c3822feea40ce (DEFAULT_TABLE table=rebalancer_dynamic_table [id=640e6e62cfc34d7081c96564110b56dd]), partition=
I20240502 14:15:04.346650 26537 tablet_service.cc:1467] Processing CreateTablet for tablet 736ed8faceec4f0fa07c62952ea32bf6 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=50fafe33ec534b21a087e7ea3eab5a91]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:04.347008 29287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32b94809b6a841cca35c3822feea40ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:04.347147 26537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 736ed8faceec4f0fa07c62952ea32bf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:04.347738 29553 tablet_service.cc:1467] Processing CreateTablet for tablet 32b94809b6a841cca35c3822feea40ce (DEFAULT_TABLE table=rebalancer_dynamic_table [id=640e6e62cfc34d7081c96564110b56dd]), partition=
I20240502 14:15:04.348176 29553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32b94809b6a841cca35c3822feea40ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:04.348738 29288 tablet_service.cc:1467] Processing CreateTablet for tablet d6c8fa23fffe491fbfcec03afdb2052a (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=50fafe33ec534b21a087e7ea3eab5a91]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:04.333180 31733 tablet_bootstrap.cc:654] T 5808efbc3ef84f3b862045c67b30cd8b P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:04.350661 26668 tablet_service.cc:1467] Processing CreateTablet for tablet d6c8fa23fffe491fbfcec03afdb2052a (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=50fafe33ec534b21a087e7ea3eab5a91]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:04.351015 29288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6c8fa23fffe491fbfcec03afdb2052a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:04.351110 26668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6c8fa23fffe491fbfcec03afdb2052a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:04.352836 29421 tablet_service.cc:1467] Processing CreateTablet for tablet 32b94809b6a841cca35c3822feea40ce (DEFAULT_TABLE table=rebalancer_dynamic_table [id=640e6e62cfc34d7081c96564110b56dd]), partition=
I20240502 14:15:04.353186 29421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32b94809b6a841cca35c3822feea40ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:04.364207 29554 tablet_service.cc:1467] Processing CreateTablet for tablet d6c8fa23fffe491fbfcec03afdb2052a (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=50fafe33ec534b21a087e7ea3eab5a91]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:04.364760 29554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6c8fa23fffe491fbfcec03afdb2052a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:04.366854 31735 tablet_bootstrap.cc:492] T 5808efbc3ef84f3b862045c67b30cd8b P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:15:04.368444 31735 tablet_bootstrap.cc:654] T 5808efbc3ef84f3b862045c67b30cd8b P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:04.374439 31734 tablet_bootstrap.cc:654] T 5808efbc3ef84f3b862045c67b30cd8b P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:04.377045 31739 tablet_bootstrap.cc:492] T 736ed8faceec4f0fa07c62952ea32bf6 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:15:04.378490 31738 tablet_bootstrap.cc:492] T 32b94809b6a841cca35c3822feea40ce P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:15:04.381031 31735 tablet_bootstrap.cc:492] T 5808efbc3ef84f3b862045c67b30cd8b P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:15:04.381227 31735 ts_tablet_manager.cc:1367] T 5808efbc3ef84f3b862045c67b30cd8b P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20240502 14:15:04.382302 31734 tablet_bootstrap.cc:492] T 5808efbc3ef84f3b862045c67b30cd8b P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:15:04.382649 31734 ts_tablet_manager.cc:1367] T 5808efbc3ef84f3b862045c67b30cd8b P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.039s user 0.006s sys 0.000s
I20240502 14:15:04.383646 31734 raft_consensus.cc:348] T 5808efbc3ef84f3b862045c67b30cd8b P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:04.384086 31734 raft_consensus.cc:374] T 5808efbc3ef84f3b862045c67b30cd8b P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:04.384261 31734 raft_consensus.cc:729] T 5808efbc3ef84f3b862045c67b30cd8b P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:15:04.384613 31734 consensus_queue.cc:260] T 5808efbc3ef84f3b862045c67b30cd8b P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:04.386273 31735 raft_consensus.cc:348] T 5808efbc3ef84f3b862045c67b30cd8b P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:04.386509 31735 raft_consensus.cc:374] T 5808efbc3ef84f3b862045c67b30cd8b P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:04.386616 31735 raft_consensus.cc:729] T 5808efbc3ef84f3b862045c67b30cd8b P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:15:04.386837 31734 ts_tablet_manager.cc:1398] T 5808efbc3ef84f3b862045c67b30cd8b P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:15:04.386863 31735 consensus_queue.cc:260] T 5808efbc3ef84f3b862045c67b30cd8b P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:04.387239 31734 tablet_bootstrap.cc:492] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:15:04.388881 31739 tablet_bootstrap.cc:654] T 736ed8faceec4f0fa07c62952ea32bf6 P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:04.389600 31738 tablet_bootstrap.cc:654] T 32b94809b6a841cca35c3822feea40ce P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:04.390003 31734 tablet_bootstrap.cc:654] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:04.397606 31735 ts_tablet_manager.cc:1398] T 5808efbc3ef84f3b862045c67b30cd8b P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240502 14:15:04.398568 31735 tablet_bootstrap.cc:492] T 736ed8faceec4f0fa07c62952ea32bf6 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:15:04.400974 31735 tablet_bootstrap.cc:654] T 736ed8faceec4f0fa07c62952ea32bf6 P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:04.407258 31739 tablet_bootstrap.cc:492] T 736ed8faceec4f0fa07c62952ea32bf6 P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:15:04.407522 31739 ts_tablet_manager.cc:1367] T 736ed8faceec4f0fa07c62952ea32bf6 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.006s
I20240502 14:15:04.408639 31739 raft_consensus.cc:348] T 736ed8faceec4f0fa07c62952ea32bf6 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:04.409263 31739 raft_consensus.cc:374] T 736ed8faceec4f0fa07c62952ea32bf6 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:04.409441 31739 raft_consensus.cc:729] T 736ed8faceec4f0fa07c62952ea32bf6 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:15:04.409567 31735 tablet_bootstrap.cc:492] T 736ed8faceec4f0fa07c62952ea32bf6 P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:15:04.409868 31735 ts_tablet_manager.cc:1367] T 736ed8faceec4f0fa07c62952ea32bf6 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20240502 14:15:04.410224 31739 consensus_queue.cc:260] T 736ed8faceec4f0fa07c62952ea32bf6 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:04.411065 31738 tablet_bootstrap.cc:492] T 32b94809b6a841cca35c3822feea40ce P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:15:04.411294 31738 ts_tablet_manager.cc:1367] T 32b94809b6a841cca35c3822feea40ce P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.033s user 0.008s sys 0.000s
I20240502 14:15:04.411330 31739 ts_tablet_manager.cc:1398] T 736ed8faceec4f0fa07c62952ea32bf6 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 14:15:04.413151 31738 raft_consensus.cc:348] T 32b94809b6a841cca35c3822feea40ce P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:04.413492 31738 raft_consensus.cc:374] T 32b94809b6a841cca35c3822feea40ce P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:04.413635 31738 raft_consensus.cc:729] T 32b94809b6a841cca35c3822feea40ce P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:15:04.413982 31738 consensus_queue.cc:260] T 32b94809b6a841cca35c3822feea40ce P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:04.417165 31738 ts_tablet_manager.cc:1398] T 32b94809b6a841cca35c3822feea40ce P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:15:04.418809 31735 raft_consensus.cc:348] T 736ed8faceec4f0fa07c62952ea32bf6 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:04.419147 31735 raft_consensus.cc:374] T 736ed8faceec4f0fa07c62952ea32bf6 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:04.419306 31735 raft_consensus.cc:729] T 736ed8faceec4f0fa07c62952ea32bf6 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:15:04.419716 31735 consensus_queue.cc:260] T 736ed8faceec4f0fa07c62952ea32bf6 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:04.420787 31735 ts_tablet_manager.cc:1398] T 736ed8faceec4f0fa07c62952ea32bf6 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240502 14:15:04.421186 31735 tablet_bootstrap.cc:492] T d6c8fa23fffe491fbfcec03afdb2052a P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:15:04.423748 31735 tablet_bootstrap.cc:654] T d6c8fa23fffe491fbfcec03afdb2052a P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:04.428588 31734 tablet_bootstrap.cc:492] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:15:04.428923 31734 ts_tablet_manager.cc:1367] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.042s user 0.002s sys 0.008s
I20240502 14:15:04.431075 31734 raft_consensus.cc:348] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:04.431751 31734 raft_consensus.cc:374] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:04.431917 31734 raft_consensus.cc:729] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:15:04.432363 31734 consensus_queue.cc:260] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:04.433665 31734 ts_tablet_manager.cc:1398] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20240502 14:15:04.436955 31734 tablet_bootstrap.cc:492] T 32b94809b6a841cca35c3822feea40ce P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:15:04.445396 31734 tablet_bootstrap.cc:654] T 32b94809b6a841cca35c3822feea40ce P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:04.450558 31620 raft_consensus.cc:482] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:04.450848 31620 raft_consensus.cc:504] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:04.452106 31620 leader_election.cc:288] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:15:04.453459 26557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736ed8faceec4f0fa07c62952ea32bf6" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:15:04.453482 29307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736ed8faceec4f0fa07c62952ea32bf6" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:15:04.453826 26557 raft_consensus.cc:2454] T 736ed8faceec4f0fa07c62952ea32bf6 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:15:04.453826 29307 raft_consensus.cc:2454] T 736ed8faceec4f0fa07c62952ea32bf6 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:15:04.454638 29510 leader_election.cc:302] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:15:04.454972 31620 raft_consensus.cc:2781] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:04.455085 31620 raft_consensus.cc:482] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:04.455199 31620 raft_consensus.cc:3037] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:04.455693 31622 raft_consensus.cc:482] T 5808efbc3ef84f3b862045c67b30cd8b P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:04.455917 31735 tablet_bootstrap.cc:492] T d6c8fa23fffe491fbfcec03afdb2052a P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:15:04.455874 31622 raft_consensus.cc:504] T 5808efbc3ef84f3b862045c67b30cd8b P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:04.456082 31735 ts_tablet_manager.cc:1367] T d6c8fa23fffe491fbfcec03afdb2052a P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.035s user 0.011s sys 0.013s
I20240502 14:15:04.457259 31622 leader_election.cc:288] T 5808efbc3ef84f3b862045c67b30cd8b P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:15:04.457864 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5808efbc3ef84f3b862045c67b30cd8b" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:15:04.459317 31620 raft_consensus.cc:504] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:04.459699 29176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5808efbc3ef84f3b862045c67b30cd8b" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:15:04.460753 26557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736ed8faceec4f0fa07c62952ea32bf6" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
W20240502 14:15:04.460804 29510 leader_election.cc:341] T 5808efbc3ef84f3b862045c67b30cd8b P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:04.461019 26557 raft_consensus.cc:3037] T 736ed8faceec4f0fa07c62952ea32bf6 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:04.461297 29309 raft_consensus.cc:2454] T 5808efbc3ef84f3b862045c67b30cd8b P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:15:04.462145 31734 tablet_bootstrap.cc:492] T 32b94809b6a841cca35c3822feea40ce P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:15:04.462785 31734 ts_tablet_manager.cc:1367] T 32b94809b6a841cca35c3822feea40ce P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.004s
I20240502 14:15:04.463817 26557 raft_consensus.cc:2454] T 736ed8faceec4f0fa07c62952ea32bf6 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:15:04.464300 29510 leader_election.cc:302] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:15:04.464686 31622 raft_consensus.cc:2781] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:04.464738 29510 leader_election.cc:302] T 5808efbc3ef84f3b862045c67b30cd8b P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, c1100e16b3b9495ca61d4f95e89b6a89; no voters: 1cb67158133e4648a6eceba32d4dbba3
I20240502 14:15:04.464968 31623 raft_consensus.cc:2781] T 5808efbc3ef84f3b862045c67b30cd8b P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:04.465135 31622 raft_consensus.cc:686] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Becoming Leader. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Running, Role: LEADER
I20240502 14:15:04.465242 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736ed8faceec4f0fa07c62952ea32bf6" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
I20240502 14:15:04.465504 29309 raft_consensus.cc:3037] T 736ed8faceec4f0fa07c62952ea32bf6 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:04.465421 31622 consensus_queue.cc:237] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:04.466609 31623 raft_consensus.cc:482] T 5808efbc3ef84f3b862045c67b30cd8b P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:04.466737 31623 raft_consensus.cc:3037] T 5808efbc3ef84f3b862045c67b30cd8b P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:04.466636 31734 raft_consensus.cc:348] T 32b94809b6a841cca35c3822feea40ce P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:04.467541 31734 raft_consensus.cc:374] T 32b94809b6a841cca35c3822feea40ce P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:04.467861 31734 raft_consensus.cc:729] T 32b94809b6a841cca35c3822feea40ce P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:15:04.468279 31734 consensus_queue.cc:260] T 32b94809b6a841cca35c3822feea40ce P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:04.469426 31734 ts_tablet_manager.cc:1398] T 32b94809b6a841cca35c3822feea40ce P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:15:04.469959 31623 raft_consensus.cc:504] T 5808efbc3ef84f3b862045c67b30cd8b P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:04.470335 31734 tablet_bootstrap.cc:492] T d6c8fa23fffe491fbfcec03afdb2052a P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:15:04.470783 31620 leader_election.cc:288] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Requested vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:15:04.470686 31623 leader_election.cc:288] T 5808efbc3ef84f3b862045c67b30cd8b P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Requested vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:15:04.473125 29309 raft_consensus.cc:2454] T 736ed8faceec4f0fa07c62952ea32bf6 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:15:04.475487 31734 tablet_bootstrap.cc:654] T d6c8fa23fffe491fbfcec03afdb2052a P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:04.477730 31735 raft_consensus.cc:348] T d6c8fa23fffe491fbfcec03afdb2052a P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:04.478511 31735 raft_consensus.cc:374] T d6c8fa23fffe491fbfcec03afdb2052a P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:04.478857 31735 raft_consensus.cc:729] T d6c8fa23fffe491fbfcec03afdb2052a P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:15:04.479529 31735 consensus_queue.cc:260] T d6c8fa23fffe491fbfcec03afdb2052a P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:04.481232 31735 ts_tablet_manager.cc:1398] T d6c8fa23fffe491fbfcec03afdb2052a P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.025s user 0.003s sys 0.000s
I20240502 14:15:04.482179 31735 tablet_bootstrap.cc:492] T 32b94809b6a841cca35c3822feea40ce P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:15:04.485548 31596 raft_consensus.cc:482] T 32b94809b6a841cca35c3822feea40ce P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:04.485742 31596 raft_consensus.cc:504] T 32b94809b6a841cca35c3822feea40ce P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:04.486755 31596 leader_election.cc:288] T 32b94809b6a841cca35c3822feea40ce P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:15:04.487519 31734 tablet_bootstrap.cc:492] T d6c8fa23fffe491fbfcec03afdb2052a P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:15:04.488509 31734 ts_tablet_manager.cc:1367] T d6c8fa23fffe491fbfcec03afdb2052a P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.000s
I20240502 14:15:04.489025 29574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32b94809b6a841cca35c3822feea40ce" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86" is_pre_election: true
I20240502 14:15:04.489320 29574 raft_consensus.cc:2454] T 32b94809b6a841cca35c3822feea40ce P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 0.
I20240502 14:15:04.489868 29379 leader_election.cc:302] T 32b94809b6a841cca35c3822feea40ce P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, 68a3c2b17ed94e06aba66ed4178132ab; no voters:
I20240502 14:15:04.490864 31595 raft_consensus.cc:2781] T 32b94809b6a841cca35c3822feea40ce P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:04.490991 31595 raft_consensus.cc:482] T 32b94809b6a841cca35c3822feea40ce P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:04.491098 31595 raft_consensus.cc:3037] T 32b94809b6a841cca35c3822feea40ce P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:04.491881 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32b94809b6a841cca35c3822feea40ce" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
W20240502 14:15:04.492640 29378 leader_election.cc:341] T 32b94809b6a841cca35c3822feea40ce P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:04.493641 31734 raft_consensus.cc:348] T d6c8fa23fffe491fbfcec03afdb2052a P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:04.493937 31734 raft_consensus.cc:374] T d6c8fa23fffe491fbfcec03afdb2052a P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:04.494017 31734 raft_consensus.cc:729] T d6c8fa23fffe491fbfcec03afdb2052a P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:15:04.494460 31595 raft_consensus.cc:504] T 32b94809b6a841cca35c3822feea40ce P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:04.495536 31734 consensus_queue.cc:260] T d6c8fa23fffe491fbfcec03afdb2052a P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:04.495920 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5808efbc3ef84f3b862045c67b30cd8b" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
I20240502 14:15:04.496094 29176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5808efbc3ef84f3b862045c67b30cd8b" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:15:04.496243 29309 raft_consensus.cc:3037] T 5808efbc3ef84f3b862045c67b30cd8b P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:04.496876 29307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32b94809b6a841cca35c3822feea40ce" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
W20240502 14:15:04.498157 29378 leader_election.cc:341] T 32b94809b6a841cca35c3822feea40ce P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:04.496208 31734 ts_tablet_manager.cc:1398] T d6c8fa23fffe491fbfcec03afdb2052a P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:15:04.501690 29309 raft_consensus.cc:2454] T 5808efbc3ef84f3b862045c67b30cd8b P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:15:04.504073 29574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32b94809b6a841cca35c3822feea40ce" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86"
I20240502 14:15:04.504369 29574 raft_consensus.cc:3037] T 32b94809b6a841cca35c3822feea40ce P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:04.507282 29510 leader_election.cc:302] T 5808efbc3ef84f3b862045c67b30cd8b P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:15:04.507685 31623 raft_consensus.cc:2781] T 5808efbc3ef84f3b862045c67b30cd8b P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:04.507874 31623 raft_consensus.cc:686] T 5808efbc3ef84f3b862045c67b30cd8b P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Becoming Leader. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Running, Role: LEADER
I20240502 14:15:04.508291 31623 consensus_queue.cc:237] T 5808efbc3ef84f3b862045c67b30cd8b P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:04.508261 29574 raft_consensus.cc:2454] T 32b94809b6a841cca35c3822feea40ce P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 1.
I20240502 14:15:04.511530 29379 leader_election.cc:302] T 32b94809b6a841cca35c3822feea40ce P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, 68a3c2b17ed94e06aba66ed4178132ab; no voters: c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:15:04.511920 31596 raft_consensus.cc:2781] T 32b94809b6a841cca35c3822feea40ce P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:04.512076 31596 raft_consensus.cc:686] T 32b94809b6a841cca35c3822feea40ce P 68a3c2b17ed94e06aba66ed4178132ab [term 1 LEADER]: Becoming Leader. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Running, Role: LEADER
I20240502 14:15:04.512336 31596 consensus_queue.cc:237] T 32b94809b6a841cca35c3822feea40ce P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
W20240502 14:15:04.514652 29510 leader_election.cc:341] T 5808efbc3ef84f3b862045c67b30cd8b P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:04.516993 31735 tablet_bootstrap.cc:654] T 32b94809b6a841cca35c3822feea40ce P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:04.521490 31595 leader_election.cc:288] T 32b94809b6a841cca35c3822feea40ce P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 election: Requested vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:15:04.523141 26281 catalog_manager.cc:5484] T 32b94809b6a841cca35c3822feea40ce P 68a3c2b17ed94e06aba66ed4178132ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199). New cstate: current_term: 1 leader_uuid: "68a3c2b17ed94e06aba66ed4178132ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:15:04.528709 31735 tablet_bootstrap.cc:492] T 32b94809b6a841cca35c3822feea40ce P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:15:04.528970 31735 ts_tablet_manager.cc:1367] T 32b94809b6a841cca35c3822feea40ce P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.047s user 0.002s sys 0.004s
I20240502 14:15:04.529881 31735 raft_consensus.cc:348] T 32b94809b6a841cca35c3822feea40ce P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:04.534009 31735 raft_consensus.cc:374] T 32b94809b6a841cca35c3822feea40ce P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:04.534560 31735 raft_consensus.cc:729] T 32b94809b6a841cca35c3822feea40ce P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:15:04.534679 31757 tablet_bootstrap.cc:492] T d6c8fa23fffe491fbfcec03afdb2052a P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:15:04.535197 31735 consensus_queue.cc:260] T 32b94809b6a841cca35c3822feea40ce P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:04.538393 26277 catalog_manager.cc:5484] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200). New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:04.538722 31735 ts_tablet_manager.cc:1398] T 32b94809b6a841cca35c3822feea40ce P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240502 14:15:04.549336 31757 tablet_bootstrap.cc:654] T d6c8fa23fffe491fbfcec03afdb2052a P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:04.560276 31757 tablet_bootstrap.cc:492] T d6c8fa23fffe491fbfcec03afdb2052a P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:15:04.561246 31757 ts_tablet_manager.cc:1367] T d6c8fa23fffe491fbfcec03afdb2052a P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.002s
I20240502 14:15:04.562700 31757 raft_consensus.cc:348] T d6c8fa23fffe491fbfcec03afdb2052a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:04.564738 31757 raft_consensus.cc:374] T d6c8fa23fffe491fbfcec03afdb2052a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:04.565137 31757 raft_consensus.cc:729] T d6c8fa23fffe491fbfcec03afdb2052a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:15:04.566354 31757 consensus_queue.cc:260] T d6c8fa23fffe491fbfcec03afdb2052a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:04.569888 31757 ts_tablet_manager.cc:1398] T d6c8fa23fffe491fbfcec03afdb2052a P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240502 14:15:04.582198 31733 tablet_bootstrap.cc:492] T 5808efbc3ef84f3b862045c67b30cd8b P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:15:04.582752 31733 ts_tablet_manager.cc:1367] T 5808efbc3ef84f3b862045c67b30cd8b P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.253s user 0.008s sys 0.092s
I20240502 14:15:04.583945 31733 raft_consensus.cc:348] T 5808efbc3ef84f3b862045c67b30cd8b P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:04.584457 31733 raft_consensus.cc:374] T 5808efbc3ef84f3b862045c67b30cd8b P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:04.584604 31733 raft_consensus.cc:729] T 5808efbc3ef84f3b862045c67b30cd8b P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:15:04.584939 31733 consensus_queue.cc:260] T 5808efbc3ef84f3b862045c67b30cd8b P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:04.585906 31733 ts_tablet_manager.cc:1398] T 5808efbc3ef84f3b862045c67b30cd8b P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:15:04.586081 26281 catalog_manager.cc:5484] T 5808efbc3ef84f3b862045c67b30cd8b P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200). New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:15:04.626649 31217 raft_consensus.cc:482] T 32b94809b6a841cca35c3822feea40ce P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:04.626842 31217 raft_consensus.cc:504] T 32b94809b6a841cca35c3822feea40ce P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:04.627700 31217 leader_election.cc:288] T 32b94809b6a841cca35c3822feea40ce P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:15:04.628492 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32b94809b6a841cca35c3822feea40ce" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:15:04.628556 29574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32b94809b6a841cca35c3822feea40ce" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86" is_pre_election: true
I20240502 14:15:04.628808 29574 raft_consensus.cc:2379] T 32b94809b6a841cca35c3822feea40ce P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c1100e16b3b9495ca61d4f95e89b6a89 in current term 1: Already voted for candidate 68a3c2b17ed94e06aba66ed4178132ab in this term.
I20240502 14:15:04.629688 29245 leader_election.cc:302] T 32b94809b6a841cca35c3822feea40ce P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c1100e16b3b9495ca61d4f95e89b6a89; no voters: 4092902b3e674857a0147f1b4f2cdb86, 68a3c2b17ed94e06aba66ed4178132ab
I20240502 14:15:04.630179 31217 raft_consensus.cc:3037] T 32b94809b6a841cca35c3822feea40ce P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:04.631152 31639 raft_consensus.cc:482] T d6c8fa23fffe491fbfcec03afdb2052a P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:04.631295 31639 raft_consensus.cc:504] T d6c8fa23fffe491fbfcec03afdb2052a P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:04.632217 31639 leader_election.cc:288] T d6c8fa23fffe491fbfcec03afdb2052a P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:15:04.633287 29574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c8fa23fffe491fbfcec03afdb2052a" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86" is_pre_election: true
I20240502 14:15:04.633606 29574 raft_consensus.cc:2454] T d6c8fa23fffe491fbfcec03afdb2052a P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 0.
I20240502 14:15:04.633841 26689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c8fa23fffe491fbfcec03afdb2052a" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:15:04.634104 26689 raft_consensus.cc:2454] T d6c8fa23fffe491fbfcec03afdb2052a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 0.
I20240502 14:15:04.634804 29245 leader_election.cc:302] T d6c8fa23fffe491fbfcec03afdb2052a P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:15:04.635247 31639 raft_consensus.cc:2781] T d6c8fa23fffe491fbfcec03afdb2052a P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:04.635367 31639 raft_consensus.cc:482] T d6c8fa23fffe491fbfcec03afdb2052a P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:04.635443 31639 raft_consensus.cc:3037] T d6c8fa23fffe491fbfcec03afdb2052a P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:04.636433 26281 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:15:04.638521 31639 raft_consensus.cc:504] T d6c8fa23fffe491fbfcec03afdb2052a P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:04.638962 31217 raft_consensus.cc:2726] T 32b94809b6a841cca35c3822feea40ce P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:15:04.641654 31639 leader_election.cc:288] T d6c8fa23fffe491fbfcec03afdb2052a P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 election: Requested vote from peers 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:15:04.643071 29574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c8fa23fffe491fbfcec03afdb2052a" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86"
I20240502 14:15:04.643344 29574 raft_consensus.cc:3037] T d6c8fa23fffe491fbfcec03afdb2052a P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:04.643307 26688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c8fa23fffe491fbfcec03afdb2052a" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:15:04.643553 26688 raft_consensus.cc:3037] T d6c8fa23fffe491fbfcec03afdb2052a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:04.647225 26688 raft_consensus.cc:2454] T d6c8fa23fffe491fbfcec03afdb2052a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 1.
I20240502 14:15:04.647915 29245 leader_election.cc:302] T d6c8fa23fffe491fbfcec03afdb2052a P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:15:04.648451 31639 raft_consensus.cc:2781] T d6c8fa23fffe491fbfcec03afdb2052a P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:04.648646 31639 raft_consensus.cc:686] T d6c8fa23fffe491fbfcec03afdb2052a P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 LEADER]: Becoming Leader. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Running, Role: LEADER
I20240502 14:15:04.647015 29574 raft_consensus.cc:2454] T d6c8fa23fffe491fbfcec03afdb2052a P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 1.
I20240502 14:15:04.649044 31639 consensus_queue.cc:237] T d6c8fa23fffe491fbfcec03afdb2052a P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:04.672447 26284 catalog_manager.cc:5484] T d6c8fa23fffe491fbfcec03afdb2052a P c1100e16b3b9495ca61d4f95e89b6a89 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198). New cstate: current_term: 1 leader_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:04.685673 29574 raft_consensus.cc:1261] T 32b94809b6a841cca35c3822feea40ce P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:04.686395 31595 consensus_queue.cc:1035] T 32b94809b6a841cca35c3822feea40ce P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:04.686647 29309 raft_consensus.cc:1261] T 32b94809b6a841cca35c3822feea40ce P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:04.687420 31595 consensus_queue.cc:1035] T 32b94809b6a841cca35c3822feea40ce P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:04.694175 26277 catalog_manager.cc:5576] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097) does not have the latest schema for tablet 32b94809b6a841cca35c3822feea40ce (table rebalancer_dynamic_table [id=640e6e62cfc34d7081c96564110b56dd]). Expected version 1 got 0
I20240502 14:15:04.695480 26285 catalog_manager.cc:5576] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659) does not have the latest schema for tablet 32b94809b6a841cca35c3822feea40ce (table rebalancer_dynamic_table [id=640e6e62cfc34d7081c96564110b56dd]). Expected version 1 got 0
I20240502 14:15:04.698748 31783 tablet.cc:1722] T 32b94809b6a841cca35c3822feea40ce P 68a3c2b17ed94e06aba66ed4178132ab: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:04.703752 31783 tablet.cc:1620] T 32b94809b6a841cca35c3822feea40ce P 68a3c2b17ed94e06aba66ed4178132ab: MemRowSet was empty: no flush needed.
I20240502 14:15:04.709311 31784 tablet.cc:1722] T 32b94809b6a841cca35c3822feea40ce P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:04.712630 31784 tablet.cc:1620] T 32b94809b6a841cca35c3822feea40ce P 4092902b3e674857a0147f1b4f2cdb86: MemRowSet was empty: no flush needed.
I20240502 14:15:04.713192 26277 catalog_manager.cc:5576] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097) does not have the latest schema for tablet 32b94809b6a841cca35c3822feea40ce (table rebalancer_dynamic_table [id=640e6e62cfc34d7081c96564110b56dd]). Expected version 1 got 0
I20240502 14:15:04.716992 31785 tablet.cc:1722] T 32b94809b6a841cca35c3822feea40ce P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:04.717864 31785 tablet.cc:1620] T 32b94809b6a841cca35c3822feea40ce P c1100e16b3b9495ca61d4f95e89b6a89: MemRowSet was empty: no flush needed.
I20240502 14:15:04.724514 26277 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=640e6e62cfc34d7081c96564110b56dd] alter complete (version 1)
I20240502 14:15:04.729007 26277 catalog_manager.cc:5576] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659) does not have the latest schema for tablet 32b94809b6a841cca35c3822feea40ce (table rebalancer_dynamic_table [id=640e6e62cfc34d7081c96564110b56dd]). Expected version 1 got 0
I20240502 14:15:04.794821 31181 raft_consensus.cc:482] T 5808efbc3ef84f3b862045c67b30cd8b P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:04.795038 31181 raft_consensus.cc:504] T 5808efbc3ef84f3b862045c67b30cd8b P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:04.795734 31181 leader_election.cc:288] T 5808efbc3ef84f3b862045c67b30cd8b P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:15:04.796291 29574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5808efbc3ef84f3b862045c67b30cd8b" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86" is_pre_election: true
I20240502 14:15:04.797240 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5808efbc3ef84f3b862045c67b30cd8b" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:15:04.797618 29309 raft_consensus.cc:2379] T 5808efbc3ef84f3b862045c67b30cd8b P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb67158133e4648a6eceba32d4dbba3 in current term 1: Already voted for candidate 4092902b3e674857a0147f1b4f2cdb86 in this term.
I20240502 14:15:04.799293 29111 leader_election.cc:302] T 5808efbc3ef84f3b862045c67b30cd8b P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3; no voters: 4092902b3e674857a0147f1b4f2cdb86, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:15:04.799652 31181 raft_consensus.cc:3037] T 5808efbc3ef84f3b862045c67b30cd8b P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:04.805270 31181 raft_consensus.cc:2726] T 5808efbc3ef84f3b862045c67b30cd8b P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:15:04.826213 26276 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:15:04.829460 29554 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:15:04.840786 31783 tablet.cc:1722] T 32b94809b6a841cca35c3822feea40ce P 68a3c2b17ed94e06aba66ed4178132ab: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:04.841514 31783 tablet.cc:1620] T 32b94809b6a841cca35c3822feea40ce P 68a3c2b17ed94e06aba66ed4178132ab: MemRowSet was empty: no flush needed.
I20240502 14:15:04.845966 31785 tablet.cc:1722] T 32b94809b6a841cca35c3822feea40ce P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:04.846534 31785 tablet.cc:1620] T 32b94809b6a841cca35c3822feea40ce P c1100e16b3b9495ca61d4f95e89b6a89: MemRowSet was empty: no flush needed.
I20240502 14:15:04.849761 31784 tablet.cc:1722] T 32b94809b6a841cca35c3822feea40ce P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:04.850327 31784 tablet.cc:1620] T 32b94809b6a841cca35c3822feea40ce P 4092902b3e674857a0147f1b4f2cdb86: MemRowSet was empty: no flush needed.
I20240502 14:15:04.864205 29156 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:04.864492 29421 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:15:04.871182 26276 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=640e6e62cfc34d7081c96564110b56dd] alter complete (version 2)
I20240502 14:15:04.891635 31623 consensus_queue.cc:1035] T 5808efbc3ef84f3b862045c67b30cd8b P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:04.895668 26407 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:04.908044 31623 consensus_queue.cc:1035] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:04.924032 31622 consensus_queue.cc:1035] T 5808efbc3ef84f3b862045c67b30cd8b P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:04.937716 26668 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:15:04.951476 31623 consensus_queue.cc:1035] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:04.982172 26277 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:15:04.984182 26537 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:04.994027 31783 tablet.cc:1722] T 32b94809b6a841cca35c3822feea40ce P 68a3c2b17ed94e06aba66ed4178132ab: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:05.000267 31784 tablet.cc:1722] T 32b94809b6a841cca35c3822feea40ce P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:05.006330 31785 tablet.cc:1722] T 32b94809b6a841cca35c3822feea40ce P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:05.018210 26797 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:05.034443 29287 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:15:05.053748 26285 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table_new_name [id=640e6e62cfc34d7081c96564110b56dd] alter complete (version 3)
I20240502 14:15:05.163569 26285 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:05.163838 26285 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:05.167629 26285 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet 32b94809b6a841cca35c3822feea40ce
I20240502 14:15:05.168969 29554 tablet_service.cc:1514] Processing DeleteTablet for tablet 32b94809b6a841cca35c3822feea40ce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:05 UTC) from {username='slave'} at 127.0.0.1:36398
I20240502 14:15:05.169348 29287 tablet_service.cc:1514] Processing DeleteTablet for tablet 32b94809b6a841cca35c3822feea40ce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:05 UTC) from {username='slave'} at 127.0.0.1:59160
I20240502 14:15:05.169806 29421 tablet_service.cc:1514] Processing DeleteTablet for tablet 32b94809b6a841cca35c3822feea40ce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:05 UTC) from {username='slave'} at 127.0.0.1:40652
I20240502 14:15:05.170276 31818 tablet_replica.cc:330] T 32b94809b6a841cca35c3822feea40ce P 4092902b3e674857a0147f1b4f2cdb86: stopping tablet replica
I20240502 14:15:05.170537 31818 raft_consensus.cc:2229] T 32b94809b6a841cca35c3822feea40ce P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:05.170825 31818 raft_consensus.cc:2258] T 32b94809b6a841cca35c3822feea40ce P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:05.172856 31820 tablet_replica.cc:330] T 32b94809b6a841cca35c3822feea40ce P 68a3c2b17ed94e06aba66ed4178132ab: stopping tablet replica
I20240502 14:15:05.173187 31820 raft_consensus.cc:2229] T 32b94809b6a841cca35c3822feea40ce P 68a3c2b17ed94e06aba66ed4178132ab [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:15:05.174022 31820 raft_consensus.cc:2258] T 32b94809b6a841cca35c3822feea40ce P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:05.174850 31819 tablet_replica.cc:330] T 32b94809b6a841cca35c3822feea40ce P c1100e16b3b9495ca61d4f95e89b6a89: stopping tablet replica
I20240502 14:15:05.175140 31818 ts_tablet_manager.cc:1875] T 32b94809b6a841cca35c3822feea40ce P 4092902b3e674857a0147f1b4f2cdb86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:05.175173 31819 raft_consensus.cc:2229] T 32b94809b6a841cca35c3822feea40ce P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:05.175452 31819 raft_consensus.cc:2258] T 32b94809b6a841cca35c3822feea40ce P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:05.177340 31819 ts_tablet_manager.cc:1875] T 32b94809b6a841cca35c3822feea40ce P c1100e16b3b9495ca61d4f95e89b6a89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:05.182037 31818 ts_tablet_manager.cc:1888] T 32b94809b6a841cca35c3822feea40ce P 4092902b3e674857a0147f1b4f2cdb86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:05.182258 31818 log.cc:1196] T 32b94809b6a841cca35c3822feea40ce P 4092902b3e674857a0147f1b4f2cdb86: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/wal/wals/32b94809b6a841cca35c3822feea40ce
I20240502 14:15:05.182732 31820 ts_tablet_manager.cc:1875] T 32b94809b6a841cca35c3822feea40ce P 68a3c2b17ed94e06aba66ed4178132ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:05.182906 31818 ts_tablet_manager.cc:1909] T 32b94809b6a841cca35c3822feea40ce P 4092902b3e674857a0147f1b4f2cdb86: Deleting consensus metadata
I20240502 14:15:05.184248 26272 catalog_manager.cc:4830] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097): tablet 32b94809b6a841cca35c3822feea40ce (table rebalancer_dynamic_table_new_name [id=640e6e62cfc34d7081c96564110b56dd]) successfully deleted
I20240502 14:15:05.187079 31819 ts_tablet_manager.cc:1888] T 32b94809b6a841cca35c3822feea40ce P c1100e16b3b9495ca61d4f95e89b6a89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:05.187326 31819 log.cc:1196] T 32b94809b6a841cca35c3822feea40ce P c1100e16b3b9495ca61d4f95e89b6a89: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal/wals/32b94809b6a841cca35c3822feea40ce
I20240502 14:15:05.187902 31819 ts_tablet_manager.cc:1909] T 32b94809b6a841cca35c3822feea40ce P c1100e16b3b9495ca61d4f95e89b6a89: Deleting consensus metadata
I20240502 14:15:05.189360 26271 catalog_manager.cc:4830] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): tablet 32b94809b6a841cca35c3822feea40ce (table rebalancer_dynamic_table_new_name [id=640e6e62cfc34d7081c96564110b56dd]) successfully deleted
I20240502 14:15:05.196431 31820 ts_tablet_manager.cc:1888] T 32b94809b6a841cca35c3822feea40ce P 68a3c2b17ed94e06aba66ed4178132ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:05.196761 31820 log.cc:1196] T 32b94809b6a841cca35c3822feea40ce P 68a3c2b17ed94e06aba66ed4178132ab: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal/wals/32b94809b6a841cca35c3822feea40ce
I20240502 14:15:05.197407 31820 ts_tablet_manager.cc:1909] T 32b94809b6a841cca35c3822feea40ce P 68a3c2b17ed94e06aba66ed4178132ab: Deleting consensus metadata
I20240502 14:15:05.199512 26272 catalog_manager.cc:4830] TS 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): tablet 32b94809b6a841cca35c3822feea40ce (table rebalancer_dynamic_table_new_name [id=640e6e62cfc34d7081c96564110b56dd]) successfully deleted
I20240502 14:15:05.240166 31639 consensus_queue.cc:1035] T d6c8fa23fffe491fbfcec03afdb2052a P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:05.249557 31217 consensus_queue.cc:1035] T d6c8fa23fffe491fbfcec03afdb2052a P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:05.255738 26285 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_extra_table_32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:05.273145 26276 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:15:05.281404 29287 tablet_service.cc:1467] Processing CreateTablet for tablet 457516f59dee4436b474af63819fad4e (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=eb9f13fa5bdc4cc3b3c074bc196dfb8c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:05.281926 29287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 457516f59dee4436b474af63819fad4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:05.284168 29421 tablet_service.cc:1467] Processing CreateTablet for tablet 457516f59dee4436b474af63819fad4e (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=eb9f13fa5bdc4cc3b3c074bc196dfb8c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:05.284724 29421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 457516f59dee4436b474af63819fad4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:05.284844 26669 tablet_service.cc:1467] Processing CreateTablet for tablet 25b56a948668442da540ca7a198f9fbb (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=eb9f13fa5bdc4cc3b3c074bc196dfb8c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:05.285274 26669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25b56a948668442da540ca7a198f9fbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:05.285167 29422 tablet_service.cc:1467] Processing CreateTablet for tablet 25b56a948668442da540ca7a198f9fbb (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=eb9f13fa5bdc4cc3b3c074bc196dfb8c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:05.285647 29422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25b56a948668442da540ca7a198f9fbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:05.288843 26668 tablet_service.cc:1467] Processing CreateTablet for tablet 2e682a5c24a64fe3ab7d325a027242d9 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=eb9f13fa5bdc4cc3b3c074bc196dfb8c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:05.289264 26668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e682a5c24a64fe3ab7d325a027242d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:05.294239 26407 tablet_service.cc:1467] Processing CreateTablet for tablet 2e682a5c24a64fe3ab7d325a027242d9 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=eb9f13fa5bdc4cc3b3c074bc196dfb8c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:05.294260 29156 tablet_service.cc:1467] Processing CreateTablet for tablet 457516f59dee4436b474af63819fad4e (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=eb9f13fa5bdc4cc3b3c074bc196dfb8c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:05.294735 29156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 457516f59dee4436b474af63819fad4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:05.294746 26407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e682a5c24a64fe3ab7d325a027242d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:05.294288 29154 tablet_service.cc:1467] Processing CreateTablet for tablet 25b56a948668442da540ca7a198f9fbb (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=eb9f13fa5bdc4cc3b3c074bc196dfb8c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:05.300334 29288 tablet_service.cc:1467] Processing CreateTablet for tablet 2e682a5c24a64fe3ab7d325a027242d9 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=eb9f13fa5bdc4cc3b3c074bc196dfb8c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:05.300757 29154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25b56a948668442da540ca7a198f9fbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:05.300805 29288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e682a5c24a64fe3ab7d325a027242d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:05.318594 29423 tablet_service.cc:1467] Processing CreateTablet for tablet 503a66003f524ac4a9d46a1d30453641 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e416087f45064e138d14371cdfe2e04f]), partition=
I20240502 14:15:05.319052 29423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 503a66003f524ac4a9d46a1d30453641. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:05.320895 26406 tablet_service.cc:1467] Processing CreateTablet for tablet 503a66003f524ac4a9d46a1d30453641 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e416087f45064e138d14371cdfe2e04f]), partition=
I20240502 14:15:05.321347 26406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 503a66003f524ac4a9d46a1d30453641. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:05.327339 26668 tablet_service.cc:1467] Processing CreateTablet for tablet 503a66003f524ac4a9d46a1d30453641 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e416087f45064e138d14371cdfe2e04f]), partition=
I20240502 14:15:05.327829 26668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 503a66003f524ac4a9d46a1d30453641. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:05.360664 31837 tablet_bootstrap.cc:492] T 25b56a948668442da540ca7a198f9fbb P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:15:05.366948 31836 tablet_bootstrap.cc:492] T 2e682a5c24a64fe3ab7d325a027242d9 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:15:05.369205 31837 tablet_bootstrap.cc:654] T 25b56a948668442da540ca7a198f9fbb P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:05.372565 31834 tablet_bootstrap.cc:492] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:15:05.373482 31841 tablet_bootstrap.cc:492] T 457516f59dee4436b474af63819fad4e P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:15:05.375725 31836 tablet_bootstrap.cc:654] T 2e682a5c24a64fe3ab7d325a027242d9 P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:05.382489 31841 tablet_bootstrap.cc:654] T 457516f59dee4436b474af63819fad4e P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:05.385048 31834 tablet_bootstrap.cc:654] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:05.385627 31835 tablet_bootstrap.cc:492] T 2e682a5c24a64fe3ab7d325a027242d9 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:15:05.387714 31836 tablet_bootstrap.cc:492] T 2e682a5c24a64fe3ab7d325a027242d9 P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:15:05.387964 31836 ts_tablet_manager.cc:1367] T 2e682a5c24a64fe3ab7d325a027242d9 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.000s
I20240502 14:15:05.389109 31836 raft_consensus.cc:348] T 2e682a5c24a64fe3ab7d325a027242d9 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:05.392181 31836 raft_consensus.cc:374] T 2e682a5c24a64fe3ab7d325a027242d9 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:05.392446 31836 raft_consensus.cc:729] T 2e682a5c24a64fe3ab7d325a027242d9 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:15:05.394042 31836 consensus_queue.cc:260] T 2e682a5c24a64fe3ab7d325a027242d9 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:05.398363 31837 tablet_bootstrap.cc:492] T 25b56a948668442da540ca7a198f9fbb P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:15:05.399191 31837 ts_tablet_manager.cc:1367] T 25b56a948668442da540ca7a198f9fbb P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.041s user 0.000s sys 0.006s
I20240502 14:15:05.400173 31837 raft_consensus.cc:348] T 25b56a948668442da540ca7a198f9fbb P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:05.400521 31837 raft_consensus.cc:374] T 25b56a948668442da540ca7a198f9fbb P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:05.400664 31837 raft_consensus.cc:729] T 25b56a948668442da540ca7a198f9fbb P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:15:05.400967 31837 consensus_queue.cc:260] T 25b56a948668442da540ca7a198f9fbb P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:05.401386 31836 ts_tablet_manager.cc:1398] T 2e682a5c24a64fe3ab7d325a027242d9 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20240502 14:15:05.403016 31837 ts_tablet_manager.cc:1398] T 25b56a948668442da540ca7a198f9fbb P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 14:15:05.403414 31837 tablet_bootstrap.cc:492] T 503a66003f524ac4a9d46a1d30453641 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:15:05.403451 31835 tablet_bootstrap.cc:654] T 2e682a5c24a64fe3ab7d325a027242d9 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:05.404350 31836 tablet_bootstrap.cc:492] T 457516f59dee4436b474af63819fad4e P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:15:05.407763 31841 tablet_bootstrap.cc:492] T 457516f59dee4436b474af63819fad4e P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:15:05.407935 31841 ts_tablet_manager.cc:1367] T 457516f59dee4436b474af63819fad4e P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.000s
I20240502 14:15:05.404510 31639 raft_consensus.cc:482] T 2e682a5c24a64fe3ab7d325a027242d9 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:05.408183 31639 raft_consensus.cc:504] T 2e682a5c24a64fe3ab7d325a027242d9 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:05.409104 31639 leader_election.cc:288] T 2e682a5c24a64fe3ab7d325a027242d9 P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:15:05.409338 31837 tablet_bootstrap.cc:654] T 503a66003f524ac4a9d46a1d30453641 P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:05.410194 26423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e682a5c24a64fe3ab7d325a027242d9" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:15:05.409024 31841 raft_consensus.cc:348] T 457516f59dee4436b474af63819fad4e P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:05.410892 26688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e682a5c24a64fe3ab7d325a027242d9" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
W20240502 14:15:05.410955 29244 leader_election.cc:341] T 2e682a5c24a64fe3ab7d325a027242d9 P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:05.411211 31841 raft_consensus.cc:374] T 457516f59dee4436b474af63819fad4e P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:05.411347 31841 raft_consensus.cc:729] T 457516f59dee4436b474af63819fad4e P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
W20240502 14:15:05.411736 29245 leader_election.cc:341] T 2e682a5c24a64fe3ab7d325a027242d9 P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:05.411710 31841 consensus_queue.cc:260] T 457516f59dee4436b474af63819fad4e P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:05.411892 29245 leader_election.cc:302] T 2e682a5c24a64fe3ab7d325a027242d9 P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c1100e16b3b9495ca61d4f95e89b6a89; no voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, 89af7f423cf44717a08d6b142f0ed49d
I20240502 14:15:05.412631 31639 raft_consensus.cc:2726] T 2e682a5c24a64fe3ab7d325a027242d9 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:15:05.415072 31836 tablet_bootstrap.cc:654] T 457516f59dee4436b474af63819fad4e P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:05.424468 31834 tablet_bootstrap.cc:492] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:15:05.424836 31834 ts_tablet_manager.cc:1367] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.052s user 0.001s sys 0.004s
I20240502 14:15:05.427269 31834 raft_consensus.cc:348] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:05.428026 31841 ts_tablet_manager.cc:1398] T 457516f59dee4436b474af63819fad4e P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.020s user 0.003s sys 0.000s
I20240502 14:15:05.428263 31834 raft_consensus.cc:374] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:05.429488 31834 raft_consensus.cc:729] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:15:05.433203 31834 consensus_queue.cc:260] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:05.434330 29575 consensus_queue.cc:237] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:05.447361 31841 tablet_bootstrap.cc:492] T 25b56a948668442da540ca7a198f9fbb P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:15:05.446576 26688 raft_consensus.cc:1261] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:05.447803 29443 raft_consensus.cc:1261] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:05.449709 31834 ts_tablet_manager.cc:1398] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20240502 14:15:05.450659 31751 consensus_queue.cc:1035] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:05.451345 31751 consensus_queue.cc:1035] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:05.451635 31834 tablet_bootstrap.cc:492] T 2e682a5c24a64fe3ab7d325a027242d9 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:15:05.460983 31835 tablet_bootstrap.cc:492] T 2e682a5c24a64fe3ab7d325a027242d9 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:15:05.461266 31835 ts_tablet_manager.cc:1367] T 2e682a5c24a64fe3ab7d325a027242d9 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.076s user 0.004s sys 0.001s
I20240502 14:15:05.462410 31835 raft_consensus.cc:348] T 2e682a5c24a64fe3ab7d325a027242d9 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:05.463409 31835 raft_consensus.cc:374] T 2e682a5c24a64fe3ab7d325a027242d9 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:05.463683 31835 raft_consensus.cc:729] T 2e682a5c24a64fe3ab7d325a027242d9 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:15:05.466622 31837 tablet_bootstrap.cc:492] T 503a66003f524ac4a9d46a1d30453641 P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:15:05.466866 31837 ts_tablet_manager.cc:1367] T 503a66003f524ac4a9d46a1d30453641 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.064s user 0.003s sys 0.010s
I20240502 14:15:05.468091 31837 raft_consensus.cc:348] T 503a66003f524ac4a9d46a1d30453641 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:05.468448 31837 raft_consensus.cc:374] T 503a66003f524ac4a9d46a1d30453641 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:05.468577 31837 raft_consensus.cc:729] T 503a66003f524ac4a9d46a1d30453641 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:15:05.469029 31837 consensus_queue.cc:260] T 503a66003f524ac4a9d46a1d30453641 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:05.469866 31837 ts_tablet_manager.cc:1398] T 503a66003f524ac4a9d46a1d30453641 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:15:05.464742 31835 consensus_queue.cc:260] T 2e682a5c24a64fe3ab7d325a027242d9 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:05.471612 31835 ts_tablet_manager.cc:1398] T 2e682a5c24a64fe3ab7d325a027242d9 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240502 14:15:05.472002 31837 tablet_bootstrap.cc:492] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:15:05.477843 31837 tablet_bootstrap.cc:654] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:05.479044 31841 tablet_bootstrap.cc:654] T 25b56a948668442da540ca7a198f9fbb P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:05.479732 31834 tablet_bootstrap.cc:654] T 2e682a5c24a64fe3ab7d325a027242d9 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:05.479961 31623 raft_consensus.cc:2932] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:05.481201 31835 tablet_bootstrap.cc:492] T 503a66003f524ac4a9d46a1d30453641 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:15:05.482857 26689 raft_consensus.cc:2932] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:05.489393 29443 raft_consensus.cc:2932] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:05.494568 31499 raft_consensus.cc:482] T 2e682a5c24a64fe3ab7d325a027242d9 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:05.494397 26285 catalog_manager.cc:5484] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:05.494769 31499 raft_consensus.cc:504] T 2e682a5c24a64fe3ab7d325a027242d9 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:05.495672 31499 leader_election.cc:288] T 2e682a5c24a64fe3ab7d325a027242d9 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:15:05.496567 29307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e682a5c24a64fe3ab7d325a027242d9" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:15:05.496829 29307 raft_consensus.cc:2454] T 2e682a5c24a64fe3ab7d325a027242d9 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:15:05.497445 26362 leader_election.cc:302] T 2e682a5c24a64fe3ab7d325a027242d9 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:15:05.498157 31499 raft_consensus.cc:2781] T 2e682a5c24a64fe3ab7d325a027242d9 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:05.498292 31499 raft_consensus.cc:482] T 2e682a5c24a64fe3ab7d325a027242d9 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:05.498389 31499 raft_consensus.cc:3037] T 2e682a5c24a64fe3ab7d325a027242d9 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:05.500612 29288 tablet_service.cc:1514] Processing DeleteTablet for tablet 1704fbfbeed04dcaac8bc663b7ee2a5b with delete_type TABLET_DATA_TOMBSTONED (TS c1100e16b3b9495ca61d4f95e89b6a89 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:59160
I20240502 14:15:05.501159 31819 tablet_replica.cc:330] T 1704fbfbeed04dcaac8bc663b7ee2a5b P c1100e16b3b9495ca61d4f95e89b6a89: stopping tablet replica
I20240502 14:15:05.501415 31819 raft_consensus.cc:2229] T 1704fbfbeed04dcaac8bc663b7ee2a5b P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:05.501669 31819 raft_consensus.cc:2258] T 1704fbfbeed04dcaac8bc663b7ee2a5b P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:05.502434 31819 ts_tablet_manager.cc:1875] T 1704fbfbeed04dcaac8bc663b7ee2a5b P c1100e16b3b9495ca61d4f95e89b6a89: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:05.503818 26689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e682a5c24a64fe3ab7d325a027242d9" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
W20240502 14:15:05.504437 26363 leader_election.cc:341] T 2e682a5c24a64fe3ab7d325a027242d9 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:05.505908 31499 raft_consensus.cc:504] T 2e682a5c24a64fe3ab7d325a027242d9 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:05.507040 29307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e682a5c24a64fe3ab7d325a027242d9" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
I20240502 14:15:05.507277 29307 raft_consensus.cc:3037] T 2e682a5c24a64fe3ab7d325a027242d9 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:05.509603 31819 ts_tablet_manager.cc:1888] T 1704fbfbeed04dcaac8bc663b7ee2a5b P c1100e16b3b9495ca61d4f95e89b6a89: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:15:05.509850 31819 log.cc:1196] T 1704fbfbeed04dcaac8bc663b7ee2a5b P c1100e16b3b9495ca61d4f95e89b6a89: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal/wals/1704fbfbeed04dcaac8bc663b7ee2a5b
I20240502 14:15:05.510190 29307 raft_consensus.cc:2454] T 2e682a5c24a64fe3ab7d325a027242d9 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:15:05.511145 26362 leader_election.cc:302] T 2e682a5c24a64fe3ab7d325a027242d9 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:15:05.511147 26271 catalog_manager.cc:4830] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): tablet 1704fbfbeed04dcaac8bc663b7ee2a5b (table rebalancer_extra_table_4 [id=052077f1ff8540afbb74e3609ab56e61]) successfully deleted
I20240502 14:15:05.511648 31628 raft_consensus.cc:2781] T 2e682a5c24a64fe3ab7d325a027242d9 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:05.515614 31499 leader_election.cc:288] T 2e682a5c24a64fe3ab7d325a027242d9 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Requested vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:15:05.516453 26689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e682a5c24a64fe3ab7d325a027242d9" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
W20240502 14:15:05.517204 26363 leader_election.cc:341] T 2e682a5c24a64fe3ab7d325a027242d9 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:05.518615 31835 tablet_bootstrap.cc:654] T 503a66003f524ac4a9d46a1d30453641 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:05.520663 31578 raft_consensus.cc:482] T 457516f59dee4436b474af63819fad4e P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:05.521088 31578 raft_consensus.cc:504] T 457516f59dee4436b474af63819fad4e P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:05.522122 31578 leader_election.cc:288] T 457516f59dee4436b474af63819fad4e P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:15:05.523803 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457516f59dee4436b474af63819fad4e" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:15:05.523356 29307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457516f59dee4436b474af63819fad4e" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
W20240502 14:15:05.524559 29111 leader_election.cc:341] T 457516f59dee4436b474af63819fad4e P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Illegal state: must be running to vote when last-logged opid is not known
W20240502 14:15:05.525578 29112 leader_election.cc:341] T 457516f59dee4436b474af63819fad4e P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:05.525846 29112 leader_election.cc:302] T 457516f59dee4436b474af63819fad4e P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:15:05.526337 31578 raft_consensus.cc:2726] T 457516f59dee4436b474af63819fad4e P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:15:05.535420 31841 tablet_bootstrap.cc:492] T 25b56a948668442da540ca7a198f9fbb P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:15:05.535854 31841 ts_tablet_manager.cc:1367] T 25b56a948668442da540ca7a198f9fbb P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.089s user 0.008s sys 0.000s
I20240502 14:15:05.538378 31834 tablet_bootstrap.cc:492] T 2e682a5c24a64fe3ab7d325a027242d9 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:15:05.539942 31628 raft_consensus.cc:686] T 2e682a5c24a64fe3ab7d325a027242d9 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Becoming Leader. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Running, Role: LEADER
I20240502 14:15:05.543700 31834 ts_tablet_manager.cc:1367] T 2e682a5c24a64fe3ab7d325a027242d9 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.092s user 0.004s sys 0.004s
I20240502 14:15:05.545167 31834 raft_consensus.cc:348] T 2e682a5c24a64fe3ab7d325a027242d9 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:05.541431 31841 raft_consensus.cc:348] T 25b56a948668442da540ca7a198f9fbb P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:05.545789 31841 raft_consensus.cc:374] T 25b56a948668442da540ca7a198f9fbb P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:05.545938 31841 raft_consensus.cc:729] T 25b56a948668442da540ca7a198f9fbb P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:15:05.546360 31841 consensus_queue.cc:260] T 25b56a948668442da540ca7a198f9fbb P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:05.548977 31834 raft_consensus.cc:374] T 2e682a5c24a64fe3ab7d325a027242d9 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:05.549235 31834 raft_consensus.cc:729] T 2e682a5c24a64fe3ab7d325a027242d9 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:15:05.554172 31834 consensus_queue.cc:260] T 2e682a5c24a64fe3ab7d325a027242d9 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:05.554795 31841 ts_tablet_manager.cc:1398] T 25b56a948668442da540ca7a198f9fbb P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20240502 14:15:05.557437 31834 ts_tablet_manager.cc:1398] T 2e682a5c24a64fe3ab7d325a027242d9 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20240502 14:15:05.565982 31834 tablet_bootstrap.cc:492] T 503a66003f524ac4a9d46a1d30453641 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:15:05.568943 31628 consensus_queue.cc:237] T 2e682a5c24a64fe3ab7d325a027242d9 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:05.569780 31835 tablet_bootstrap.cc:492] T 503a66003f524ac4a9d46a1d30453641 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:15:05.570120 31835 ts_tablet_manager.cc:1367] T 503a66003f524ac4a9d46a1d30453641 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.089s user 0.003s sys 0.003s
I20240502 14:15:05.571290 31835 raft_consensus.cc:348] T 503a66003f524ac4a9d46a1d30453641 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:05.571666 31835 raft_consensus.cc:374] T 503a66003f524ac4a9d46a1d30453641 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:05.571787 31835 raft_consensus.cc:729] T 503a66003f524ac4a9d46a1d30453641 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:15:05.572105 31835 consensus_queue.cc:260] T 503a66003f524ac4a9d46a1d30453641 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:05.572983 31835 ts_tablet_manager.cc:1398] T 503a66003f524ac4a9d46a1d30453641 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:15:05.573858 31836 tablet_bootstrap.cc:492] T 457516f59dee4436b474af63819fad4e P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:15:05.574173 31836 ts_tablet_manager.cc:1367] T 457516f59dee4436b474af63819fad4e P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.170s user 0.008s sys 0.002s
I20240502 14:15:05.574259 31834 tablet_bootstrap.cc:654] T 503a66003f524ac4a9d46a1d30453641 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:05.575094 31836 raft_consensus.cc:348] T 457516f59dee4436b474af63819fad4e P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:05.582916 31836 raft_consensus.cc:374] T 457516f59dee4436b474af63819fad4e P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:05.586423 26285 catalog_manager.cc:5484] T 2e682a5c24a64fe3ab7d325a027242d9 P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: term changed from 0 to 1, leader changed from <none> to 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193). New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:05.583277 31836 raft_consensus.cc:729] T 457516f59dee4436b474af63819fad4e P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:15:05.588660 31836 consensus_queue.cc:260] T 457516f59dee4436b474af63819fad4e P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:05.598682 31836 ts_tablet_manager.cc:1398] T 457516f59dee4436b474af63819fad4e P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20240502 14:15:05.612573 31837 tablet_bootstrap.cc:492] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:15:05.612962 31837 ts_tablet_manager.cc:1367] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.141s user 0.015s sys 0.014s
I20240502 14:15:05.616950 31837 raft_consensus.cc:348] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:05.618434 31837 raft_consensus.cc:374] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:05.618592 31837 raft_consensus.cc:729] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:15:05.618875 31837 consensus_queue.cc:260] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:05.620720 31837 ts_tablet_manager.cc:1398] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:15:05.626201 31834 tablet_bootstrap.cc:492] T 503a66003f524ac4a9d46a1d30453641 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:15:05.626727 31834 ts_tablet_manager.cc:1367] T 503a66003f524ac4a9d46a1d30453641 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.061s user 0.012s sys 0.019s
I20240502 14:15:05.627976 31834 raft_consensus.cc:348] T 503a66003f524ac4a9d46a1d30453641 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:05.630254 31834 raft_consensus.cc:374] T 503a66003f524ac4a9d46a1d30453641 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:05.630434 31834 raft_consensus.cc:729] T 503a66003f524ac4a9d46a1d30453641 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:15:05.630848 31834 consensus_queue.cc:260] T 503a66003f524ac4a9d46a1d30453641 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:05.636940 31834 ts_tablet_manager.cc:1398] T 503a66003f524ac4a9d46a1d30453641 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240502 14:15:05.650149 31596 raft_consensus.cc:482] T 503a66003f524ac4a9d46a1d30453641 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:05.650359 31596 raft_consensus.cc:504] T 503a66003f524ac4a9d46a1d30453641 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:05.651427 31596 leader_election.cc:288] T 503a66003f524ac4a9d46a1d30453641 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:15:05.653257 26423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "503a66003f524ac4a9d46a1d30453641" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:15:05.653515 26423 raft_consensus.cc:2454] T 503a66003f524ac4a9d46a1d30453641 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 0.
I20240502 14:15:05.655563 29378 leader_election.cc:302] T 503a66003f524ac4a9d46a1d30453641 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:15:05.658517 31595 raft_consensus.cc:2781] T 503a66003f524ac4a9d46a1d30453641 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:05.658656 31595 raft_consensus.cc:482] T 503a66003f524ac4a9d46a1d30453641 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:05.658749 31595 raft_consensus.cc:3037] T 503a66003f524ac4a9d46a1d30453641 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:05.661757 31595 raft_consensus.cc:504] T 503a66003f524ac4a9d46a1d30453641 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:05.663230 26423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "503a66003f524ac4a9d46a1d30453641" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:15:05.663669 26423 raft_consensus.cc:3037] T 503a66003f524ac4a9d46a1d30453641 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:05.666388 26689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "503a66003f524ac4a9d46a1d30453641" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:15:05.666625 26689 raft_consensus.cc:2454] T 503a66003f524ac4a9d46a1d30453641 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 0.
I20240502 14:15:05.666777 26423 raft_consensus.cc:2454] T 503a66003f524ac4a9d46a1d30453641 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 1.
I20240502 14:15:05.666957 26688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "503a66003f524ac4a9d46a1d30453641" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:15:05.667167 26688 raft_consensus.cc:3037] T 503a66003f524ac4a9d46a1d30453641 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:05.669117 29378 leader_election.cc:302] T 503a66003f524ac4a9d46a1d30453641 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:15:05.669488 31596 raft_consensus.cc:2781] T 503a66003f524ac4a9d46a1d30453641 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:05.669870 31595 leader_election.cc:288] T 503a66003f524ac4a9d46a1d30453641 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 election: Requested vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:15:05.670163 26688 raft_consensus.cc:2454] T 503a66003f524ac4a9d46a1d30453641 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 1.
I20240502 14:15:05.677328 31596 raft_consensus.cc:686] T 503a66003f524ac4a9d46a1d30453641 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 LEADER]: Becoming Leader. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Running, Role: LEADER
I20240502 14:15:05.677691 31596 consensus_queue.cc:237] T 503a66003f524ac4a9d46a1d30453641 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:05.684911 31617 raft_consensus.cc:482] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:05.685127 31617 raft_consensus.cc:504] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:05.686019 31617 leader_election.cc:288] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:15:05.686825 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25b56a948668442da540ca7a198f9fbb" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:15:05.687125 29443 raft_consensus.cc:2454] T 25b56a948668442da540ca7a198f9fbb P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:15:05.688081 26625 leader_election.cc:302] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters:
I20240502 14:15:05.688498 31617 raft_consensus.cc:2781] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:05.688614 31617 raft_consensus.cc:482] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:05.688695 31617 raft_consensus.cc:3037] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:05.691785 31617 raft_consensus.cc:504] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:05.697482 29440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25b56a948668442da540ca7a198f9fbb" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
I20240502 14:15:05.697757 29440 raft_consensus.cc:3037] T 25b56a948668442da540ca7a198f9fbb P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:05.701135 31617 leader_election.cc:288] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Requested vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:15:05.703331 29176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25b56a948668442da540ca7a198f9fbb" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:15:05.703576 29176 raft_consensus.cc:3037] T 25b56a948668442da540ca7a198f9fbb P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:05.703454 26285 catalog_manager.cc:5484] T 503a66003f524ac4a9d46a1d30453641 P 68a3c2b17ed94e06aba66ed4178132ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199). New cstate: current_term: 1 leader_uuid: "68a3c2b17ed94e06aba66ed4178132ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:05.706569 29176 raft_consensus.cc:2454] T 25b56a948668442da540ca7a198f9fbb P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:15:05.708120 26624 leader_election.cc:302] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters:
I20240502 14:15:05.708571 31881 raft_consensus.cc:2781] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:05.708688 31881 raft_consensus.cc:686] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Becoming Leader. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Running, Role: LEADER
I20240502 14:15:05.708993 31881 consensus_queue.cc:237] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:05.709730 29174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25b56a948668442da540ca7a198f9fbb" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:15:05.709996 29174 raft_consensus.cc:2362] T 25b56a948668442da540ca7a198f9fbb P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1. Re-sending same reply.
I20240502 14:15:05.712992 29440 raft_consensus.cc:2454] T 25b56a948668442da540ca7a198f9fbb P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:15:05.724797 26285 catalog_manager.cc:5484] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195). New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:05.736793 31596 raft_consensus.cc:482] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:05.736999 31596 raft_consensus.cc:504] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:05.737847 31596 leader_election.cc:288] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:15:05.738580 29307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457516f59dee4436b474af63819fad4e" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:15:05.738643 29176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457516f59dee4436b474af63819fad4e" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:15:05.738853 29307 raft_consensus.cc:2454] T 457516f59dee4436b474af63819fad4e P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 0.
I20240502 14:15:05.738965 29176 raft_consensus.cc:2454] T 457516f59dee4436b474af63819fad4e P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 0.
I20240502 14:15:05.739382 29378 leader_election.cc:302] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:15:05.739665 31596 raft_consensus.cc:2781] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:05.739807 31596 raft_consensus.cc:482] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:05.739903 31596 raft_consensus.cc:3037] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:05.742619 31596 raft_consensus.cc:504] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:05.744195 29307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457516f59dee4436b474af63819fad4e" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
I20240502 14:15:05.744534 29307 raft_consensus.cc:3037] T 457516f59dee4436b474af63819fad4e P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:05.752496 29307 raft_consensus.cc:2454] T 457516f59dee4436b474af63819fad4e P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 1.
I20240502 14:15:05.756107 29378 leader_election.cc:302] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:15:05.756721 31877 raft_consensus.cc:2781] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:05.756918 31877 raft_consensus.cc:686] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab [term 1 LEADER]: Becoming Leader. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Running, Role: LEADER
I20240502 14:15:05.757318 31877 consensus_queue.cc:237] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:05.761704 31596 leader_election.cc:288] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 election: Requested vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:15:05.763360 29176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457516f59dee4436b474af63819fad4e" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:15:05.763638 29176 raft_consensus.cc:3037] T 457516f59dee4436b474af63819fad4e P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:05.764501 26285 catalog_manager.cc:5484] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199). New cstate: current_term: 1 leader_uuid: "68a3c2b17ed94e06aba66ed4178132ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:05.767786 29176 raft_consensus.cc:2454] T 457516f59dee4436b474af63819fad4e P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 1.
I20240502 14:15:05.878679 26285 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:15:05.910708 26688 raft_consensus.cc:1261] T 503a66003f524ac4a9d46a1d30453641 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:05.911655 31596 consensus_queue.cc:1035] T 503a66003f524ac4a9d46a1d30453641 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:05.924379 26423 raft_consensus.cc:1261] T 503a66003f524ac4a9d46a1d30453641 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:05.925431 31596 consensus_queue.cc:1035] T 503a66003f524ac4a9d46a1d30453641 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:05.935990 26276 catalog_manager.cc:5576] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325) does not have the latest schema for tablet 503a66003f524ac4a9d46a1d30453641 (table rebalancer_dynamic_table [id=e416087f45064e138d14371cdfe2e04f]). Expected version 1 got 0
I20240502 14:15:05.938346 26276 catalog_manager.cc:5576] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985) does not have the latest schema for tablet 503a66003f524ac4a9d46a1d30453641 (table rebalancer_dynamic_table [id=e416087f45064e138d14371cdfe2e04f]). Expected version 1 got 0
I20240502 14:15:05.941318 31922 tablet.cc:1722] T 503a66003f524ac4a9d46a1d30453641 P 68a3c2b17ed94e06aba66ed4178132ab: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:05.943584 31922 tablet.cc:1620] T 503a66003f524ac4a9d46a1d30453641 P 68a3c2b17ed94e06aba66ed4178132ab: MemRowSet was empty: no flush needed.
I20240502 14:15:05.945497 31923 tablet.cc:1722] T 503a66003f524ac4a9d46a1d30453641 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:05.946210 31923 tablet.cc:1620] T 503a66003f524ac4a9d46a1d30453641 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: MemRowSet was empty: no flush needed.
I20240502 14:15:05.946609 31924 tablet.cc:1722] T 503a66003f524ac4a9d46a1d30453641 P 89af7f423cf44717a08d6b142f0ed49d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:05.947110 26276 catalog_manager.cc:5576] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985) does not have the latest schema for tablet 503a66003f524ac4a9d46a1d30453641 (table rebalancer_dynamic_table [id=e416087f45064e138d14371cdfe2e04f]). Expected version 1 got 0
I20240502 14:15:05.947454 31924 tablet.cc:1620] T 503a66003f524ac4a9d46a1d30453641 P 89af7f423cf44717a08d6b142f0ed49d: MemRowSet was empty: no flush needed.
I20240502 14:15:05.953370 26276 catalog_manager.cc:5576] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325) does not have the latest schema for tablet 503a66003f524ac4a9d46a1d30453641 (table rebalancer_dynamic_table [id=e416087f45064e138d14371cdfe2e04f]). Expected version 1 got 0
I20240502 14:15:05.955662 26283 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=e416087f45064e138d14371cdfe2e04f] alter complete (version 1)
I20240502 14:15:05.965406 31628 consensus_queue.cc:1035] T 2e682a5c24a64fe3ab7d325a027242d9 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:05.976857 29156 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:06.007766 29421 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:15:06.019445 29554 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:15:06.029196 26688 raft_consensus.cc:3037] T 2e682a5c24a64fe3ab7d325a027242d9 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:06.048581 31617 raft_consensus.cc:482] T 2e682a5c24a64fe3ab7d325a027242d9 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:06.048789 31617 raft_consensus.cc:504] T 2e682a5c24a64fe3ab7d325a027242d9 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:06.049531 31617 leader_election.cc:288] T 2e682a5c24a64fe3ab7d325a027242d9 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:15:06.050412 26423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e682a5c24a64fe3ab7d325a027242d9" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:15:06.051229 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e682a5c24a64fe3ab7d325a027242d9" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:15:06.054057 26624 leader_election.cc:302] T 2e682a5c24a64fe3ab7d325a027242d9 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters: 89af7f423cf44717a08d6b142f0ed49d, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:15:06.054545 31617 raft_consensus.cc:2726] T 2e682a5c24a64fe3ab7d325a027242d9 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:15:06.059266 26276 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:15:06.063882 31628 consensus_queue.cc:1035] T 2e682a5c24a64fe3ab7d325a027242d9 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:06.081084 31922 tablet.cc:1722] T 503a66003f524ac4a9d46a1d30453641 P 68a3c2b17ed94e06aba66ed4178132ab: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:06.081640 31922 tablet.cc:1620] T 503a66003f524ac4a9d46a1d30453641 P 68a3c2b17ed94e06aba66ed4178132ab: MemRowSet was empty: no flush needed.
I20240502 14:15:06.086156 31923 tablet.cc:1722] T 503a66003f524ac4a9d46a1d30453641 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:06.086676 31923 tablet.cc:1620] T 503a66003f524ac4a9d46a1d30453641 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: MemRowSet was empty: no flush needed.
I20240502 14:15:06.087199 31924 tablet.cc:1722] T 503a66003f524ac4a9d46a1d30453641 P 89af7f423cf44717a08d6b142f0ed49d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:06.087703 31924 tablet.cc:1620] T 503a66003f524ac4a9d46a1d30453641 P 89af7f423cf44717a08d6b142f0ed49d: MemRowSet was empty: no flush needed.
I20240502 14:15:06.095810 26276 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=e416087f45064e138d14371cdfe2e04f] alter complete (version 2)
I20240502 14:15:06.125186 26537 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:06.125515 29288 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:15:06.139210 26668 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:15:06.145710 26407 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:06.168138 26797 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:06.194528 31617 consensus_queue.cc:1035] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:06.201506 26283 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:15:06.223862 31922 tablet.cc:1722] T 503a66003f524ac4a9d46a1d30453641 P 68a3c2b17ed94e06aba66ed4178132ab: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:06.228158 31923 tablet.cc:1722] T 503a66003f524ac4a9d46a1d30453641 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:06.238775 26276 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table_new_name [id=e416087f45064e138d14371cdfe2e04f] alter complete (version 3)
I20240502 14:15:06.240042 31924 tablet.cc:1722] T 503a66003f524ac4a9d46a1d30453641 P 89af7f423cf44717a08d6b142f0ed49d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:06.257207 31881 consensus_queue.cc:1035] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:06.285787 26276 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_extra_table_34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:06.297835 26537 tablet_service.cc:1467] Processing CreateTablet for tablet aa4c1d9bf17a48b1b657bb7d03409efb (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=b4142465f8a74e2baf1a95d85f3fd781]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:06.297969 29156 tablet_service.cc:1467] Processing CreateTablet for tablet aa4c1d9bf17a48b1b657bb7d03409efb (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=b4142465f8a74e2baf1a95d85f3fd781]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:06.298331 26537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa4c1d9bf17a48b1b657bb7d03409efb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:06.298218 29554 tablet_service.cc:1467] Processing CreateTablet for tablet aa4c1d9bf17a48b1b657bb7d03409efb (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=b4142465f8a74e2baf1a95d85f3fd781]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:06.298408 29156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa4c1d9bf17a48b1b657bb7d03409efb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:06.298663 29554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa4c1d9bf17a48b1b657bb7d03409efb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:06.303799 26538 tablet_service.cc:1467] Processing CreateTablet for tablet 451b303a32a3404d8d584203d6704b35 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=b4142465f8a74e2baf1a95d85f3fd781]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:06.304220 26538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 451b303a32a3404d8d584203d6704b35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:06.307333 29154 tablet_service.cc:1467] Processing CreateTablet for tablet 451b303a32a3404d8d584203d6704b35 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=b4142465f8a74e2baf1a95d85f3fd781]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:06.307780 29154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 451b303a32a3404d8d584203d6704b35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:06.308236 26668 tablet_service.cc:1467] Processing CreateTablet for tablet 451b303a32a3404d8d584203d6704b35 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=b4142465f8a74e2baf1a95d85f3fd781]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:06.308367 29288 tablet_service.cc:1467] Processing CreateTablet for tablet c115987981694996abe797e03b27f799 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=b4142465f8a74e2baf1a95d85f3fd781]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:06.308740 26668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 451b303a32a3404d8d584203d6704b35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:06.308820 29288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c115987981694996abe797e03b27f799. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:06.314126 29155 tablet_service.cc:1467] Processing CreateTablet for tablet c115987981694996abe797e03b27f799 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=b4142465f8a74e2baf1a95d85f3fd781]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:06.314612 29155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c115987981694996abe797e03b27f799. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:06.316177 29553 tablet_service.cc:1467] Processing CreateTablet for tablet c115987981694996abe797e03b27f799 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=b4142465f8a74e2baf1a95d85f3fd781]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:06.317602 29553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c115987981694996abe797e03b27f799. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:06.343701 31949 tablet_bootstrap.cc:492] T c115987981694996abe797e03b27f799 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:15:06.344452 31951 tablet_bootstrap.cc:492] T aa4c1d9bf17a48b1b657bb7d03409efb P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:15:06.351693 31949 tablet_bootstrap.cc:654] T c115987981694996abe797e03b27f799 P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:06.352305 31951 tablet_bootstrap.cc:654] T aa4c1d9bf17a48b1b657bb7d03409efb P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:06.362576 31595 consensus_queue.cc:1035] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:06.371590 26276 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:06.371800 26276 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:06.371904 31952 tablet_bootstrap.cc:492] T 451b303a32a3404d8d584203d6704b35 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:15:06.375372 26276 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet 503a66003f524ac4a9d46a1d30453641
I20240502 14:15:06.376706 26669 tablet_service.cc:1514] Processing DeleteTablet for tablet 503a66003f524ac4a9d46a1d30453641 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:06 UTC) from {username='slave'} at 127.0.0.1:34150
I20240502 14:15:06.377166 26407 tablet_service.cc:1514] Processing DeleteTablet for tablet 503a66003f524ac4a9d46a1d30453641 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:06 UTC) from {username='slave'} at 127.0.0.1:37422
I20240502 14:15:06.377337 29421 tablet_service.cc:1514] Processing DeleteTablet for tablet 503a66003f524ac4a9d46a1d30453641 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:06 UTC) from {username='slave'} at 127.0.0.1:40652
I20240502 14:15:06.407632 31950 tablet_bootstrap.cc:492] T 451b303a32a3404d8d584203d6704b35 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:15:06.409794 31950 tablet_bootstrap.cc:654] T 451b303a32a3404d8d584203d6704b35 P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:06.411684 31953 tablet_bootstrap.cc:492] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:15:06.415588 31959 tablet_replica.cc:330] T 503a66003f524ac4a9d46a1d30453641 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: stopping tablet replica
I20240502 14:15:06.415977 31959 raft_consensus.cc:2229] T 503a66003f524ac4a9d46a1d30453641 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:06.416369 31959 raft_consensus.cc:2258] T 503a66003f524ac4a9d46a1d30453641 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:06.423897 31951 tablet_bootstrap.cc:492] T aa4c1d9bf17a48b1b657bb7d03409efb P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:15:06.424276 31951 ts_tablet_manager.cc:1367] T aa4c1d9bf17a48b1b657bb7d03409efb P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.080s user 0.005s sys 0.002s
I20240502 14:15:06.425644 31951 raft_consensus.cc:348] T aa4c1d9bf17a48b1b657bb7d03409efb P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:06.426501 31951 raft_consensus.cc:374] T aa4c1d9bf17a48b1b657bb7d03409efb P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:06.426673 31951 raft_consensus.cc:729] T aa4c1d9bf17a48b1b657bb7d03409efb P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:15:06.427094 31951 consensus_queue.cc:260] T aa4c1d9bf17a48b1b657bb7d03409efb P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:06.429440 31595 consensus_queue.cc:1035] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:06.439787 31959 ts_tablet_manager.cc:1875] T 503a66003f524ac4a9d46a1d30453641 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:06.440145 31623 raft_consensus.cc:482] T aa4c1d9bf17a48b1b657bb7d03409efb P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:06.440454 31623 raft_consensus.cc:504] T aa4c1d9bf17a48b1b657bb7d03409efb P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:06.441397 31623 leader_election.cc:288] T aa4c1d9bf17a48b1b657bb7d03409efb P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:15:06.442243 29175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa4c1d9bf17a48b1b657bb7d03409efb" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
W20240502 14:15:06.443270 29510 leader_election.cc:341] T aa4c1d9bf17a48b1b657bb7d03409efb P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:06.443238 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa4c1d9bf17a48b1b657bb7d03409efb" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
W20240502 14:15:06.443965 29510 leader_election.cc:341] T aa4c1d9bf17a48b1b657bb7d03409efb P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:06.444099 29510 leader_election.cc:302] T aa4c1d9bf17a48b1b657bb7d03409efb P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86; no voters: 1cb67158133e4648a6eceba32d4dbba3, ba187e38a2754d60a12de9ac4856430a
I20240502 14:15:06.444334 31949 tablet_bootstrap.cc:492] T c115987981694996abe797e03b27f799 P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:15:06.444541 31949 ts_tablet_manager.cc:1367] T c115987981694996abe797e03b27f799 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.101s user 0.016s sys 0.052s
I20240502 14:15:06.446080 31623 raft_consensus.cc:2726] T aa4c1d9bf17a48b1b657bb7d03409efb P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:15:06.446003 31949 raft_consensus.cc:348] T c115987981694996abe797e03b27f799 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:06.446224 31949 raft_consensus.cc:374] T c115987981694996abe797e03b27f799 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:06.446331 31949 raft_consensus.cc:729] T c115987981694996abe797e03b27f799 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:15:06.446589 31949 consensus_queue.cc:260] T c115987981694996abe797e03b27f799 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:06.447167 31949 ts_tablet_manager.cc:1398] T c115987981694996abe797e03b27f799 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:15:06.451634 31953 tablet_bootstrap.cc:654] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:06.457000 31960 tablet_replica.cc:330] T 503a66003f524ac4a9d46a1d30453641 P 89af7f423cf44717a08d6b142f0ed49d: stopping tablet replica
I20240502 14:15:06.457098 31951 ts_tablet_manager.cc:1398] T aa4c1d9bf17a48b1b657bb7d03409efb P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.033s user 0.002s sys 0.000s
I20240502 14:15:06.457247 31960 raft_consensus.cc:2229] T 503a66003f524ac4a9d46a1d30453641 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:06.457444 31960 raft_consensus.cc:2258] T 503a66003f524ac4a9d46a1d30453641 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:06.458935 31951 tablet_bootstrap.cc:492] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:15:06.460150 31960 ts_tablet_manager.cc:1875] T 503a66003f524ac4a9d46a1d30453641 P 89af7f423cf44717a08d6b142f0ed49d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:06.461249 31951 tablet_bootstrap.cc:654] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:06.470204 31951 tablet_bootstrap.cc:492] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:15:06.470495 31951 ts_tablet_manager.cc:1367] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.006s
I20240502 14:15:06.471462 31951 raft_consensus.cc:348] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:06.471793 31951 raft_consensus.cc:374] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:06.471931 31951 raft_consensus.cc:729] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:15:06.472260 31951 consensus_queue.cc:260] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:06.473109 31951 ts_tablet_manager.cc:1398] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:06.478260 26276 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:15:06.483872 31961 tablet_replica.cc:330] T 503a66003f524ac4a9d46a1d30453641 P 68a3c2b17ed94e06aba66ed4178132ab: stopping tablet replica
I20240502 14:15:06.484205 31961 raft_consensus.cc:2229] T 503a66003f524ac4a9d46a1d30453641 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:15:06.484681 31961 raft_consensus.cc:2258] T 503a66003f524ac4a9d46a1d30453641 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:06.486270 31952 tablet_bootstrap.cc:654] T 451b303a32a3404d8d584203d6704b35 P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:06.487957 29422 tablet_service.cc:1467] Processing CreateTablet for tablet ac1a42217d3a49d18b61b0422e164187 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=34496715169f4212a8fd4453b575574e]), partition=
I20240502 14:15:06.488365 29422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac1a42217d3a49d18b61b0422e164187. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:06.488449 29554 tablet_service.cc:1467] Processing CreateTablet for tablet ac1a42217d3a49d18b61b0422e164187 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=34496715169f4212a8fd4453b575574e]), partition=
I20240502 14:15:06.488914 29554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac1a42217d3a49d18b61b0422e164187. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:06.498311 31950 tablet_bootstrap.cc:492] T 451b303a32a3404d8d584203d6704b35 P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:15:06.500308 31950 ts_tablet_manager.cc:1367] T 451b303a32a3404d8d584203d6704b35 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.093s user 0.004s sys 0.000s
I20240502 14:15:06.500556 31623 raft_consensus.cc:482] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:06.500764 31623 raft_consensus.cc:504] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:06.501372 31623 leader_election.cc:288] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:15:06.502166 29175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c115987981694996abe797e03b27f799" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
W20240502 14:15:06.503015 29510 leader_election.cc:341] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:06.504163 29307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c115987981694996abe797e03b27f799" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:15:06.504411 29307 raft_consensus.cc:2454] T c115987981694996abe797e03b27f799 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:15:06.504822 29510 leader_election.cc:302] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, c1100e16b3b9495ca61d4f95e89b6a89; no voters: 1cb67158133e4648a6eceba32d4dbba3
I20240502 14:15:06.505307 31950 raft_consensus.cc:348] T 451b303a32a3404d8d584203d6704b35 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:06.505481 31623 raft_consensus.cc:2781] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:06.505578 31623 raft_consensus.cc:482] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:06.505666 31623 raft_consensus.cc:3037] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:06.508064 31961 ts_tablet_manager.cc:1875] T 503a66003f524ac4a9d46a1d30453641 P 68a3c2b17ed94e06aba66ed4178132ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:06.508214 31623 raft_consensus.cc:504] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:06.509404 31623 leader_election.cc:288] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Requested vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:15:06.510001 29307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c115987981694996abe797e03b27f799" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
I20240502 14:15:06.510251 29307 raft_consensus.cc:3037] T c115987981694996abe797e03b27f799 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:06.513258 29307 raft_consensus.cc:2454] T c115987981694996abe797e03b27f799 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:15:06.514250 29510 leader_election.cc:302] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:15:06.514611 31623 raft_consensus.cc:2781] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:06.515206 31951 tablet_bootstrap.cc:492] T ac1a42217d3a49d18b61b0422e164187 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:15:06.515157 31623 raft_consensus.cc:686] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Becoming Leader. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Running, Role: LEADER
I20240502 14:15:06.517990 31951 tablet_bootstrap.cc:654] T ac1a42217d3a49d18b61b0422e164187 P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
W20240502 14:15:06.518934 26755 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 155117us
I20240502 14:15:06.521570 26797 tablet_service.cc:1467] Processing CreateTablet for tablet ac1a42217d3a49d18b61b0422e164187 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=34496715169f4212a8fd4453b575574e]), partition=
I20240502 14:15:06.522034 26797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac1a42217d3a49d18b61b0422e164187. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:06.538341 31959 ts_tablet_manager.cc:1888] T 503a66003f524ac4a9d46a1d30453641 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:06.541836 31959 log.cc:1196] T 503a66003f524ac4a9d46a1d30453641 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/wals/503a66003f524ac4a9d46a1d30453641
I20240502 14:15:06.553682 31959 ts_tablet_manager.cc:1909] T 503a66003f524ac4a9d46a1d30453641 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting consensus metadata
I20240502 14:15:06.554970 26272 catalog_manager.cc:4830] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): tablet 503a66003f524ac4a9d46a1d30453641 (table rebalancer_dynamic_table_new_name [id=e416087f45064e138d14371cdfe2e04f]) successfully deleted
I20240502 14:15:06.556092 31623 consensus_queue.cc:237] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:06.557272 31950 raft_consensus.cc:374] T 451b303a32a3404d8d584203d6704b35 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:06.557375 31950 raft_consensus.cc:729] T 451b303a32a3404d8d584203d6704b35 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:15:06.557614 31950 consensus_queue.cc:260] T 451b303a32a3404d8d584203d6704b35 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:06.560873 31950 ts_tablet_manager.cc:1398] T 451b303a32a3404d8d584203d6704b35 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.060s user 0.000s sys 0.002s
I20240502 14:15:06.561589 31950 tablet_bootstrap.cc:492] T aa4c1d9bf17a48b1b657bb7d03409efb P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
W20240502 14:15:06.569339 26625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.196:32843, user_credentials={real_user=slave}} blocked reactor thread for 44252.6us
I20240502 14:15:06.570824 29174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c115987981694996abe797e03b27f799" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
W20240502 14:15:06.571659 29510 leader_election.cc:341] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:06.574103 31952 tablet_bootstrap.cc:492] T 451b303a32a3404d8d584203d6704b35 P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:15:06.574725 31952 ts_tablet_manager.cc:1367] T 451b303a32a3404d8d584203d6704b35 P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.203s user 0.005s sys 0.000s
I20240502 14:15:06.575570 31953 tablet_bootstrap.cc:492] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:15:06.575832 31953 ts_tablet_manager.cc:1367] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.164s user 0.002s sys 0.003s
I20240502 14:15:06.575703 31952 raft_consensus.cc:348] T 451b303a32a3404d8d584203d6704b35 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:06.575981 31952 raft_consensus.cc:374] T 451b303a32a3404d8d584203d6704b35 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:06.575976 31960 ts_tablet_manager.cc:1888] T 503a66003f524ac4a9d46a1d30453641 P 89af7f423cf44717a08d6b142f0ed49d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:06.576107 31960 log.cc:1196] T 503a66003f524ac4a9d46a1d30453641 P 89af7f423cf44717a08d6b142f0ed49d: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/503a66003f524ac4a9d46a1d30453641
I20240502 14:15:06.576092 31952 raft_consensus.cc:729] T 451b303a32a3404d8d584203d6704b35 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:15:06.576537 31960 ts_tablet_manager.cc:1909] T 503a66003f524ac4a9d46a1d30453641 P 89af7f423cf44717a08d6b142f0ed49d: Deleting consensus metadata
I20240502 14:15:06.577114 31953 raft_consensus.cc:348] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:06.579077 31953 raft_consensus.cc:374] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:06.579234 31953 raft_consensus.cc:729] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:15:06.579524 31953 consensus_queue.cc:260] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:06.581367 31953 ts_tablet_manager.cc:1398] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:15:06.583240 26271 catalog_manager.cc:4830] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): tablet 503a66003f524ac4a9d46a1d30453641 (table rebalancer_dynamic_table_new_name [id=e416087f45064e138d14371cdfe2e04f]) successfully deleted
I20240502 14:15:06.586104 31950 tablet_bootstrap.cc:654] T aa4c1d9bf17a48b1b657bb7d03409efb P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:06.576366 31952 consensus_queue.cc:260] T 451b303a32a3404d8d584203d6704b35 P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:06.595048 31952 ts_tablet_manager.cc:1398] T 451b303a32a3404d8d584203d6704b35 P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20240502 14:15:06.597121 31952 tablet_bootstrap.cc:492] T aa4c1d9bf17a48b1b657bb7d03409efb P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:15:06.599025 31967 tablet_bootstrap.cc:492] T ac1a42217d3a49d18b61b0422e164187 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:15:06.599354 31961 ts_tablet_manager.cc:1888] T 503a66003f524ac4a9d46a1d30453641 P 68a3c2b17ed94e06aba66ed4178132ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:06.607730 31961 log.cc:1196] T 503a66003f524ac4a9d46a1d30453641 P 68a3c2b17ed94e06aba66ed4178132ab: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal/wals/503a66003f524ac4a9d46a1d30453641
I20240502 14:15:06.609311 31950 tablet_bootstrap.cc:492] T aa4c1d9bf17a48b1b657bb7d03409efb P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:15:06.609479 31950 ts_tablet_manager.cc:1367] T aa4c1d9bf17a48b1b657bb7d03409efb P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.048s user 0.001s sys 0.002s
I20240502 14:15:06.610357 31950 raft_consensus.cc:348] T aa4c1d9bf17a48b1b657bb7d03409efb P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:06.610586 31950 raft_consensus.cc:374] T aa4c1d9bf17a48b1b657bb7d03409efb P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:06.610649 31950 raft_consensus.cc:729] T aa4c1d9bf17a48b1b657bb7d03409efb P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:15:06.611364 31952 tablet_bootstrap.cc:654] T aa4c1d9bf17a48b1b657bb7d03409efb P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:06.601045 31967 tablet_bootstrap.cc:654] T ac1a42217d3a49d18b61b0422e164187 P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:06.612061 31961 ts_tablet_manager.cc:1909] T 503a66003f524ac4a9d46a1d30453641 P 68a3c2b17ed94e06aba66ed4178132ab: Deleting consensus metadata
I20240502 14:15:06.616204 26272 catalog_manager.cc:4830] TS 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): tablet 503a66003f524ac4a9d46a1d30453641 (table rebalancer_dynamic_table_new_name [id=e416087f45064e138d14371cdfe2e04f]) successfully deleted
I20240502 14:15:06.616887 31950 consensus_queue.cc:260] T aa4c1d9bf17a48b1b657bb7d03409efb P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:06.617493 31950 ts_tablet_manager.cc:1398] T aa4c1d9bf17a48b1b657bb7d03409efb P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240502 14:15:06.628055 31977 tablet_bootstrap.cc:492] T ac1a42217d3a49d18b61b0422e164187 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:15:06.632429 26277 catalog_manager.cc:5484] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200). New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:15:06.638788 31967 tablet_bootstrap.cc:492] T ac1a42217d3a49d18b61b0422e164187 P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:15:06.639091 31967 ts_tablet_manager.cc:1367] T ac1a42217d3a49d18b61b0422e164187 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.004s
I20240502 14:15:06.640137 31967 raft_consensus.cc:348] T ac1a42217d3a49d18b61b0422e164187 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:06.640519 31967 raft_consensus.cc:374] T ac1a42217d3a49d18b61b0422e164187 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:06.640676 31967 raft_consensus.cc:729] T ac1a42217d3a49d18b61b0422e164187 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:15:06.640934 31967 consensus_queue.cc:260] T ac1a42217d3a49d18b61b0422e164187 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
W20240502 14:15:06.629670 29112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.196:32843, user_credentials={real_user=slave}} blocked reactor thread for 98752.9us
I20240502 14:15:06.644685 31967 ts_tablet_manager.cc:1398] T ac1a42217d3a49d18b61b0422e164187 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
W20240502 14:15:06.647730 29511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.199:41781, user_credentials={real_user=slave}} blocked reactor thread for 119015us
I20240502 14:15:06.654511 31977 tablet_bootstrap.cc:654] T ac1a42217d3a49d18b61b0422e164187 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:06.676741 31977 tablet_bootstrap.cc:492] T ac1a42217d3a49d18b61b0422e164187 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:15:06.677059 31977 ts_tablet_manager.cc:1367] T ac1a42217d3a49d18b61b0422e164187 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.049s user 0.011s sys 0.012s
I20240502 14:15:06.678133 31977 raft_consensus.cc:348] T ac1a42217d3a49d18b61b0422e164187 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:06.678519 31977 raft_consensus.cc:374] T ac1a42217d3a49d18b61b0422e164187 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:06.678684 31977 raft_consensus.cc:729] T ac1a42217d3a49d18b61b0422e164187 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:15:06.679071 31977 consensus_queue.cc:260] T ac1a42217d3a49d18b61b0422e164187 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:06.679972 31977 ts_tablet_manager.cc:1398] T ac1a42217d3a49d18b61b0422e164187 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:15:06.680459 31951 tablet_bootstrap.cc:492] T ac1a42217d3a49d18b61b0422e164187 P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:15:06.680939 31951 ts_tablet_manager.cc:1367] T ac1a42217d3a49d18b61b0422e164187 P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.166s user 0.015s sys 0.003s
I20240502 14:15:06.682405 31952 tablet_bootstrap.cc:492] T aa4c1d9bf17a48b1b657bb7d03409efb P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:15:06.683002 31952 ts_tablet_manager.cc:1367] T aa4c1d9bf17a48b1b657bb7d03409efb P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.086s user 0.007s sys 0.000s
I20240502 14:15:06.688268 31952 raft_consensus.cc:348] T aa4c1d9bf17a48b1b657bb7d03409efb P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:06.688802 31951 raft_consensus.cc:348] T ac1a42217d3a49d18b61b0422e164187 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:06.689100 31951 raft_consensus.cc:374] T ac1a42217d3a49d18b61b0422e164187 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:06.689244 31951 raft_consensus.cc:729] T ac1a42217d3a49d18b61b0422e164187 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:15:06.689575 31951 consensus_queue.cc:260] T ac1a42217d3a49d18b61b0422e164187 P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:06.696884 31951 ts_tablet_manager.cc:1398] T ac1a42217d3a49d18b61b0422e164187 P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.015s user 0.000s sys 0.002s
I20240502 14:15:06.697117 31952 raft_consensus.cc:374] T aa4c1d9bf17a48b1b657bb7d03409efb P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:06.697278 31952 raft_consensus.cc:729] T aa4c1d9bf17a48b1b657bb7d03409efb P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:15:06.697583 31952 consensus_queue.cc:260] T aa4c1d9bf17a48b1b657bb7d03409efb P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:06.699366 31952 ts_tablet_manager.cc:1398] T aa4c1d9bf17a48b1b657bb7d03409efb P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240502 14:15:06.699834 31952 tablet_bootstrap.cc:492] T c115987981694996abe797e03b27f799 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:15:06.711171 31952 tablet_bootstrap.cc:654] T c115987981694996abe797e03b27f799 P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:06.715430 26558 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c586354e10cb4bedbbb20bd103ad477e"
dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42560
I20240502 14:15:06.715904 26558 raft_consensus.cc:595] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Received request to transfer leadership
I20240502 14:15:06.717191 31617 raft_consensus.cc:482] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:06.717406 31617 raft_consensus.cc:504] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:06.718245 31617 leader_election.cc:288] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:15:06.718715 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "451b303a32a3404d8d584203d6704b35" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:15:06.718784 29174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "451b303a32a3404d8d584203d6704b35" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:15:06.718928 26558 raft_consensus.cc:2454] T 451b303a32a3404d8d584203d6704b35 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:15:06.718987 29174 raft_consensus.cc:2454] T 451b303a32a3404d8d584203d6704b35 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:15:06.719385 26624 leader_election.cc:302] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:15:06.721562 31617 raft_consensus.cc:2781] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:06.721659 31617 raft_consensus.cc:482] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:06.721758 31617 raft_consensus.cc:3037] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:06.721817 31952 tablet_bootstrap.cc:492] T c115987981694996abe797e03b27f799 P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:15:06.721968 31952 ts_tablet_manager.cc:1367] T c115987981694996abe797e03b27f799 P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20240502 14:15:06.723073 31952 raft_consensus.cc:348] T c115987981694996abe797e03b27f799 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:06.723412 31952 raft_consensus.cc:374] T c115987981694996abe797e03b27f799 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:06.723517 31952 raft_consensus.cc:729] T c115987981694996abe797e03b27f799 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:15:06.723780 31971 raft_consensus.cc:482] T ac1a42217d3a49d18b61b0422e164187 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:06.723784 31952 consensus_queue.cc:260] T c115987981694996abe797e03b27f799 P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:06.723928 31971 raft_consensus.cc:504] T ac1a42217d3a49d18b61b0422e164187 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:06.724824 31971 leader_election.cc:288] T ac1a42217d3a49d18b61b0422e164187 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:15:06.728005 31749 raft_consensus.cc:979] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a: : Instructing follower e4cfb8fc5a474177af6848a6efa13a27 to start an election
I20240502 14:15:06.728132 31749 raft_consensus.cc:1067] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Signalling peer e4cfb8fc5a474177af6848a6efa13a27 to start an election
I20240502 14:15:06.729142 26817 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c586354e10cb4bedbbb20bd103ad477e"
dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
from {username='slave'} at 127.19.94.194:53513
I20240502 14:15:06.729466 26817 raft_consensus.cc:482] T c586354e10cb4bedbbb20bd103ad477e P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:06.729606 26817 raft_consensus.cc:3037] T c586354e10cb4bedbbb20bd103ad477e P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:06.730566 26819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac1a42217d3a49d18b61b0422e164187" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:15:06.730850 26819 raft_consensus.cc:2454] T ac1a42217d3a49d18b61b0422e164187 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:15:06.731276 31956 raft_consensus.cc:482] T ac1a42217d3a49d18b61b0422e164187 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:06.731426 31956 raft_consensus.cc:504] T ac1a42217d3a49d18b61b0422e164187 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:06.731662 29511 leader_election.cc:302] T ac1a42217d3a49d18b61b0422e164187 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:15:06.732161 31956 leader_election.cc:288] T ac1a42217d3a49d18b61b0422e164187 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:15:06.732460 31623 raft_consensus.cc:2781] T ac1a42217d3a49d18b61b0422e164187 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:06.732563 31623 raft_consensus.cc:482] T ac1a42217d3a49d18b61b0422e164187 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:06.732661 31623 raft_consensus.cc:3037] T ac1a42217d3a49d18b61b0422e164187 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:06.733160 29442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac1a42217d3a49d18b61b0422e164187" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:15:06.733428 29442 raft_consensus.cc:2454] T ac1a42217d3a49d18b61b0422e164187 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:15:06.736238 29575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac1a42217d3a49d18b61b0422e164187" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86" is_pre_election: true
I20240502 14:15:06.735462 31623 raft_consensus.cc:504] T ac1a42217d3a49d18b61b0422e164187 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:06.739513 29575 raft_consensus.cc:2379] T ac1a42217d3a49d18b61b0422e164187 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4cfb8fc5a474177af6848a6efa13a27 in current term 1: Already voted for candidate 4092902b3e674857a0147f1b4f2cdb86 in this term.
I20240502 14:15:06.740650 29442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac1a42217d3a49d18b61b0422e164187" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:15:06.740963 29442 raft_consensus.cc:2454] T ac1a42217d3a49d18b61b0422e164187 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 0.
I20240502 14:15:06.741642 26755 leader_election.cc:302] T ac1a42217d3a49d18b61b0422e164187 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, e4cfb8fc5a474177af6848a6efa13a27; no voters: 4092902b3e674857a0147f1b4f2cdb86
I20240502 14:15:06.742332 31957 raft_consensus.cc:2781] T ac1a42217d3a49d18b61b0422e164187 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:06.742451 31957 raft_consensus.cc:482] T ac1a42217d3a49d18b61b0422e164187 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:06.742558 31957 raft_consensus.cc:3037] T ac1a42217d3a49d18b61b0422e164187 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:06.742902 31617 raft_consensus.cc:504] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:06.744500 26819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac1a42217d3a49d18b61b0422e164187" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:15:06.745486 31623 leader_election.cc:288] T ac1a42217d3a49d18b61b0422e164187 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Requested vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:15:06.746191 31957 raft_consensus.cc:504] T ac1a42217d3a49d18b61b0422e164187 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:06.747615 29440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac1a42217d3a49d18b61b0422e164187" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
I20240502 14:15:06.747713 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "451b303a32a3404d8d584203d6704b35" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:15:06.747888 29440 raft_consensus.cc:3037] T ac1a42217d3a49d18b61b0422e164187 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:06.747900 26558 raft_consensus.cc:3037] T 451b303a32a3404d8d584203d6704b35 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:06.750808 29440 raft_consensus.cc:2454] T ac1a42217d3a49d18b61b0422e164187 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
I20240502 14:15:06.751289 26558 raft_consensus.cc:2454] T 451b303a32a3404d8d584203d6704b35 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:15:06.751794 26624 leader_election.cc:302] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:15:06.752127 31969 raft_consensus.cc:2781] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:06.752312 31969 raft_consensus.cc:686] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Becoming Leader. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Running, Role: LEADER
I20240502 14:15:06.752619 31969 consensus_queue.cc:237] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:06.753257 31617 leader_election.cc:288] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Requested vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:15:06.754122 31957 leader_election.cc:288] T ac1a42217d3a49d18b61b0422e164187 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Requested vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:15:06.754541 29176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "451b303a32a3404d8d584203d6704b35" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:15:06.754750 29176 raft_consensus.cc:3037] T 451b303a32a3404d8d584203d6704b35 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:06.754855 29442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac1a42217d3a49d18b61b0422e164187" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
I20240502 14:15:06.755185 29442 raft_consensus.cc:2379] T ac1a42217d3a49d18b61b0422e164187 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4092902b3e674857a0147f1b4f2cdb86 in current term 1: Already voted for candidate e4cfb8fc5a474177af6848a6efa13a27 in this term.
I20240502 14:15:06.756110 26819 raft_consensus.cc:2379] T ac1a42217d3a49d18b61b0422e164187 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4092902b3e674857a0147f1b4f2cdb86 in current term 1: Already voted for candidate e4cfb8fc5a474177af6848a6efa13a27 in this term.
I20240502 14:15:06.757030 29511 leader_election.cc:302] T ac1a42217d3a49d18b61b0422e164187 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86; no voters: 68a3c2b17ed94e06aba66ed4178132ab, e4cfb8fc5a474177af6848a6efa13a27
I20240502 14:15:06.757423 31623 raft_consensus.cc:2726] T ac1a42217d3a49d18b61b0422e164187 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 14:15:06.758005 26755 leader_election.cc:302] T ac1a42217d3a49d18b61b0422e164187 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:15:06.757917 29176 raft_consensus.cc:2454] T 451b303a32a3404d8d584203d6704b35 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:15:06.760255 31956 raft_consensus.cc:2781] T ac1a42217d3a49d18b61b0422e164187 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:06.760537 31956 raft_consensus.cc:686] T ac1a42217d3a49d18b61b0422e164187 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Becoming Leader. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Running, Role: LEADER
I20240502 14:15:06.760969 31956 consensus_queue.cc:237] T ac1a42217d3a49d18b61b0422e164187 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:06.766436 29575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac1a42217d3a49d18b61b0422e164187" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86"
I20240502 14:15:06.766852 29575 raft_consensus.cc:2379] T ac1a42217d3a49d18b61b0422e164187 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e4cfb8fc5a474177af6848a6efa13a27 in current term 1: Already voted for candidate 4092902b3e674857a0147f1b4f2cdb86 in this term.
I20240502 14:15:06.769853 26277 catalog_manager.cc:5484] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195). New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:06.784618 26817 raft_consensus.cc:504] T c586354e10cb4bedbbb20bd103ad477e P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:06.786343 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c586354e10cb4bedbbb20bd103ad477e" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:15:06.786572 26558 raft_consensus.cc:3032] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:06.786643 26558 raft_consensus.cc:729] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Running, Role: LEADER
I20240502 14:15:06.786964 26558 consensus_queue.cc:260] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:06.787576 26558 raft_consensus.cc:3037] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:06.787948 31952 ts_tablet_manager.cc:1398] T c115987981694996abe797e03b27f799 P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.066s user 0.002s sys 0.000s
I20240502 14:15:06.795266 26277 catalog_manager.cc:5484] T ac1a42217d3a49d18b61b0422e164187 P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196). New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:15:06.796125 26558 raft_consensus.cc:2454] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 2.
I20240502 14:15:06.801928 26817 leader_election.cc:288] T c586354e10cb4bedbbb20bd103ad477e P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 election: Requested vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:15:06.803030 26689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c586354e10cb4bedbbb20bd103ad477e" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:15:06.803040 29575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c586354e10cb4bedbbb20bd103ad477e" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4092902b3e674857a0147f1b4f2cdb86"
I20240502 14:15:06.803287 26689 raft_consensus.cc:3037] T c586354e10cb4bedbbb20bd103ad477e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:06.803329 29575 raft_consensus.cc:3037] T c586354e10cb4bedbbb20bd103ad477e P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:06.808432 29575 raft_consensus.cc:2454] T c586354e10cb4bedbbb20bd103ad477e P 4092902b3e674857a0147f1b4f2cdb86 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 2.
I20240502 14:15:06.809013 26755 leader_election.cc:302] T c586354e10cb4bedbbb20bd103ad477e P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, ba187e38a2754d60a12de9ac4856430a, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:15:06.810149 26689 raft_consensus.cc:2454] T c586354e10cb4bedbbb20bd103ad477e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 2.
I20240502 14:15:06.810460 31956 raft_consensus.cc:2781] T c586354e10cb4bedbbb20bd103ad477e P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:06.810863 31956 raft_consensus.cc:686] T c586354e10cb4bedbbb20bd103ad477e P e4cfb8fc5a474177af6848a6efa13a27 [term 2 LEADER]: Becoming Leader. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Running, Role: LEADER
I20240502 14:15:06.811196 31956 consensus_queue.cc:237] T c586354e10cb4bedbbb20bd103ad477e P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:06.821739 26277 catalog_manager.cc:5484] T c586354e10cb4bedbbb20bd103ad477e P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: term changed from 1 to 2, leader changed from ba187e38a2754d60a12de9ac4856430a (127.19.94.194) to e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196). New cstate: current_term: 2 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:15:06.961047 31917 raft_consensus.cc:482] T c115987981694996abe797e03b27f799 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:06.961297 31917 raft_consensus.cc:504] T c115987981694996abe797e03b27f799 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:06.962582 31917 leader_election.cc:288] T c115987981694996abe797e03b27f799 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:15:06.963222 31369 raft_consensus.cc:482] T aa4c1d9bf17a48b1b657bb7d03409efb P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:06.963428 31369 raft_consensus.cc:504] T aa4c1d9bf17a48b1b657bb7d03409efb P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:06.964412 31369 leader_election.cc:288] T aa4c1d9bf17a48b1b657bb7d03409efb P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:15:06.964901 29575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c115987981694996abe797e03b27f799" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86" is_pre_election: true
I20240502 14:15:06.967192 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c115987981694996abe797e03b27f799" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:15:06.967607 29309 raft_consensus.cc:2379] T c115987981694996abe797e03b27f799 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cb67158133e4648a6eceba32d4dbba3 in current term 1: Already voted for candidate 4092902b3e674857a0147f1b4f2cdb86 in this term.
I20240502 14:15:06.968499 29176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa4c1d9bf17a48b1b657bb7d03409efb" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:15:06.969069 29176 raft_consensus.cc:2454] T aa4c1d9bf17a48b1b657bb7d03409efb P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 0.
I20240502 14:15:06.970140 29111 leader_election.cc:302] T c115987981694996abe797e03b27f799 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3; no voters: 4092902b3e674857a0147f1b4f2cdb86, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:15:06.970984 26493 leader_election.cc:302] T aa4c1d9bf17a48b1b657bb7d03409efb P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:15:06.971812 29575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa4c1d9bf17a48b1b657bb7d03409efb" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86" is_pre_election: true
I20240502 14:15:06.972411 29575 raft_consensus.cc:2454] T aa4c1d9bf17a48b1b657bb7d03409efb P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 0.
I20240502 14:15:06.977149 31369 raft_consensus.cc:2781] T aa4c1d9bf17a48b1b657bb7d03409efb P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:06.977319 31369 raft_consensus.cc:482] T aa4c1d9bf17a48b1b657bb7d03409efb P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:06.977413 31369 raft_consensus.cc:3037] T aa4c1d9bf17a48b1b657bb7d03409efb P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:06.980990 31917 raft_consensus.cc:3037] T c115987981694996abe797e03b27f799 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:06.983515 26276 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:15:07.006356 31917 raft_consensus.cc:2726] T c115987981694996abe797e03b27f799 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:15:07.006297 31369 raft_consensus.cc:504] T aa4c1d9bf17a48b1b657bb7d03409efb P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:07.012004 29176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa4c1d9bf17a48b1b657bb7d03409efb" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:15:07.012418 29176 raft_consensus.cc:3037] T aa4c1d9bf17a48b1b657bb7d03409efb P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:07.031100 29442 raft_consensus.cc:1261] T ac1a42217d3a49d18b61b0422e164187 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:07.039542 31956 consensus_queue.cc:1035] T ac1a42217d3a49d18b61b0422e164187 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:07.053198 29575 raft_consensus.cc:1261] T ac1a42217d3a49d18b61b0422e164187 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:07.054114 31957 consensus_queue.cc:1035] T ac1a42217d3a49d18b61b0422e164187 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:07.063220 26285 catalog_manager.cc:5576] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097) does not have the latest schema for tablet ac1a42217d3a49d18b61b0422e164187 (table rebalancer_dynamic_table [id=34496715169f4212a8fd4453b575574e]). Expected version 1 got 0
I20240502 14:15:07.018229 29176 raft_consensus.cc:2454] T aa4c1d9bf17a48b1b657bb7d03409efb P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:15:07.064325 31369 leader_election.cc:288] T aa4c1d9bf17a48b1b657bb7d03409efb P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Requested vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:15:07.066355 31971 consensus_queue.cc:1035] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:07.068449 29575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa4c1d9bf17a48b1b657bb7d03409efb" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86"
I20240502 14:15:07.068732 29575 raft_consensus.cc:3037] T aa4c1d9bf17a48b1b657bb7d03409efb P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:07.081636 31973 consensus_queue.cc:1035] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:07.087950 26493 leader_election.cc:302] T aa4c1d9bf17a48b1b657bb7d03409efb P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:15:07.088527 31369 raft_consensus.cc:2781] T aa4c1d9bf17a48b1b657bb7d03409efb P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:07.088733 31369 raft_consensus.cc:686] T aa4c1d9bf17a48b1b657bb7d03409efb P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Becoming Leader. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Running, Role: LEADER
I20240502 14:15:07.089207 31369 consensus_queue.cc:237] T aa4c1d9bf17a48b1b657bb7d03409efb P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:07.094938 32034 tablet.cc:1722] T ac1a42217d3a49d18b61b0422e164187 P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:07.098745 26278 catalog_manager.cc:5484] T aa4c1d9bf17a48b1b657bb7d03409efb P ba187e38a2754d60a12de9ac4856430a reported cstate change: term changed from 0 to 1, leader changed from <none> to ba187e38a2754d60a12de9ac4856430a (127.19.94.194). New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:15:07.099514 26276 catalog_manager.cc:5576] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097) does not have the latest schema for tablet ac1a42217d3a49d18b61b0422e164187 (table rebalancer_dynamic_table [id=34496715169f4212a8fd4453b575574e]). Expected version 1 got 0
I20240502 14:15:07.100589 32034 tablet.cc:1620] T ac1a42217d3a49d18b61b0422e164187 P 4092902b3e674857a0147f1b4f2cdb86: MemRowSet was empty: no flush needed.
I20240502 14:15:07.118721 29575 raft_consensus.cc:2454] T aa4c1d9bf17a48b1b657bb7d03409efb P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:15:07.142217 26285 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=34496715169f4212a8fd4453b575574e] alter complete (version 1)
I20240502 14:15:07.154273 32031 tablet.cc:1722] T ac1a42217d3a49d18b61b0422e164187 P e4cfb8fc5a474177af6848a6efa13a27: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:07.159299 32031 tablet.cc:1620] T ac1a42217d3a49d18b61b0422e164187 P e4cfb8fc5a474177af6848a6efa13a27: MemRowSet was empty: no flush needed.
I20240502 14:15:07.165122 26278 catalog_manager.cc:5576] TS 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781) does not have the latest schema for tablet ac1a42217d3a49d18b61b0422e164187 (table rebalancer_dynamic_table [id=34496715169f4212a8fd4453b575574e]). Expected version 1 got 0
I20240502 14:15:07.175061 32049 tablet.cc:1722] T ac1a42217d3a49d18b61b0422e164187 P 68a3c2b17ed94e06aba66ed4178132ab: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:07.176079 32049 tablet.cc:1620] T ac1a42217d3a49d18b61b0422e164187 P 68a3c2b17ed94e06aba66ed4178132ab: MemRowSet was empty: no flush needed.
I20240502 14:15:07.177953 26278 catalog_manager.cc:5576] TS 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781) does not have the latest schema for tablet ac1a42217d3a49d18b61b0422e164187 (table rebalancer_dynamic_table [id=34496715169f4212a8fd4453b575574e]). Expected version 1 got 0
I20240502 14:15:07.187005 31617 consensus_queue.cc:1035] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:07.224515 29154 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:07.237246 29554 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:15:07.243956 31881 consensus_queue.cc:1035] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:07.249730 26557 raft_consensus.cc:1261] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a [term 2 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:15:07.250403 31957 consensus_queue.cc:1035] T c586354e10cb4bedbbb20bd103ad477e P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:07.258327 26278 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:15:07.262988 29422 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:15:07.271976 32031 tablet.cc:1722] T ac1a42217d3a49d18b61b0422e164187 P e4cfb8fc5a474177af6848a6efa13a27: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:07.272541 32031 tablet.cc:1620] T ac1a42217d3a49d18b61b0422e164187 P e4cfb8fc5a474177af6848a6efa13a27: MemRowSet was empty: no flush needed.
I20240502 14:15:07.273141 32034 tablet.cc:1722] T ac1a42217d3a49d18b61b0422e164187 P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:07.273623 32034 tablet.cc:1620] T ac1a42217d3a49d18b61b0422e164187 P 4092902b3e674857a0147f1b4f2cdb86: MemRowSet was empty: no flush needed.
I20240502 14:15:07.274070 32049 tablet.cc:1722] T ac1a42217d3a49d18b61b0422e164187 P 68a3c2b17ed94e06aba66ed4178132ab: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:07.274616 32049 tablet.cc:1620] T ac1a42217d3a49d18b61b0422e164187 P 68a3c2b17ed94e06aba66ed4178132ab: MemRowSet was empty: no flush needed.
I20240502 14:15:07.286942 26285 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=34496715169f4212a8fd4453b575574e] alter complete (version 2)
I20240502 14:15:07.304255 29575 raft_consensus.cc:1261] T c586354e10cb4bedbbb20bd103ad477e P 4092902b3e674857a0147f1b4f2cdb86 [term 2 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:15:07.305454 31192 consensus_queue.cc:1035] T c586354e10cb4bedbbb20bd103ad477e P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:07.308828 26669 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:15:07.319422 26688 raft_consensus.cc:1261] T c586354e10cb4bedbbb20bd103ad477e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 2 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:15:07.320020 31192 consensus_queue.cc:1035] T c586354e10cb4bedbbb20bd103ad477e P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:07.332474 26407 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:07.357561 26538 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:07.384563 29288 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:15:07.394840 26285 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:15:07.405335 32031 tablet.cc:1722] T ac1a42217d3a49d18b61b0422e164187 P e4cfb8fc5a474177af6848a6efa13a27: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:07.406172 32034 tablet.cc:1722] T ac1a42217d3a49d18b61b0422e164187 P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:07.418864 26285 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table_new_name [id=34496715169f4212a8fd4453b575574e] alter complete (version 3)
I20240502 14:15:07.425427 32049 tablet.cc:1722] T ac1a42217d3a49d18b61b0422e164187 P 68a3c2b17ed94e06aba66ed4178132ab: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:07.433147 26797 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:15:07.522925 26278 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:07.523237 26278 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:07.531292 26278 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet ac1a42217d3a49d18b61b0422e164187
I20240502 14:15:07.532476 26797 tablet_service.cc:1514] Processing DeleteTablet for tablet ac1a42217d3a49d18b61b0422e164187 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:07 UTC) from {username='slave'} at 127.0.0.1:56010
I20240502 14:15:07.532652 29422 tablet_service.cc:1514] Processing DeleteTablet for tablet ac1a42217d3a49d18b61b0422e164187 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:07 UTC) from {username='slave'} at 127.0.0.1:40652
I20240502 14:15:07.533216 29554 tablet_service.cc:1514] Processing DeleteTablet for tablet ac1a42217d3a49d18b61b0422e164187 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:07 UTC) from {username='slave'} at 127.0.0.1:36398
I20240502 14:15:07.534420 32072 tablet_replica.cc:330] T ac1a42217d3a49d18b61b0422e164187 P 4092902b3e674857a0147f1b4f2cdb86: stopping tablet replica
I20240502 14:15:07.535072 32072 raft_consensus.cc:2229] T ac1a42217d3a49d18b61b0422e164187 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:07.535508 32072 raft_consensus.cc:2258] T ac1a42217d3a49d18b61b0422e164187 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:07.535619 31369 consensus_queue.cc:1035] T aa4c1d9bf17a48b1b657bb7d03409efb P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:07.537158 32072 ts_tablet_manager.cc:1875] T ac1a42217d3a49d18b61b0422e164187 P 4092902b3e674857a0147f1b4f2cdb86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:07.545482 32072 ts_tablet_manager.cc:1888] T ac1a42217d3a49d18b61b0422e164187 P 4092902b3e674857a0147f1b4f2cdb86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:07.545737 32072 log.cc:1196] T ac1a42217d3a49d18b61b0422e164187 P 4092902b3e674857a0147f1b4f2cdb86: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/wal/wals/ac1a42217d3a49d18b61b0422e164187
I20240502 14:15:07.546329 32072 ts_tablet_manager.cc:1909] T ac1a42217d3a49d18b61b0422e164187 P 4092902b3e674857a0147f1b4f2cdb86: Deleting consensus metadata
I20240502 14:15:07.547770 26272 catalog_manager.cc:4830] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097): tablet ac1a42217d3a49d18b61b0422e164187 (table rebalancer_dynamic_table_new_name [id=34496715169f4212a8fd4453b575574e]) successfully deleted
I20240502 14:15:07.553985 32070 tablet_replica.cc:330] T ac1a42217d3a49d18b61b0422e164187 P e4cfb8fc5a474177af6848a6efa13a27: stopping tablet replica
I20240502 14:15:07.554417 32070 raft_consensus.cc:2229] T ac1a42217d3a49d18b61b0422e164187 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:15:07.554889 32070 raft_consensus.cc:2258] T ac1a42217d3a49d18b61b0422e164187 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:07.556360 32071 tablet_replica.cc:330] T ac1a42217d3a49d18b61b0422e164187 P 68a3c2b17ed94e06aba66ed4178132ab: stopping tablet replica
I20240502 14:15:07.559753 31749 consensus_queue.cc:1035] T aa4c1d9bf17a48b1b657bb7d03409efb P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:07.560197 32071 raft_consensus.cc:2229] T ac1a42217d3a49d18b61b0422e164187 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:07.561197 32071 raft_consensus.cc:2258] T ac1a42217d3a49d18b61b0422e164187 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:07.559923 32070 ts_tablet_manager.cc:1875] T ac1a42217d3a49d18b61b0422e164187 P e4cfb8fc5a474177af6848a6efa13a27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:07.563642 32071 ts_tablet_manager.cc:1875] T ac1a42217d3a49d18b61b0422e164187 P 68a3c2b17ed94e06aba66ed4178132ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:07.568848 32070 ts_tablet_manager.cc:1888] T ac1a42217d3a49d18b61b0422e164187 P e4cfb8fc5a474177af6848a6efa13a27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:07.569077 32070 log.cc:1196] T ac1a42217d3a49d18b61b0422e164187 P e4cfb8fc5a474177af6848a6efa13a27: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/wals/ac1a42217d3a49d18b61b0422e164187
I20240502 14:15:07.569641 32070 ts_tablet_manager.cc:1909] T ac1a42217d3a49d18b61b0422e164187 P e4cfb8fc5a474177af6848a6efa13a27: Deleting consensus metadata
I20240502 14:15:07.570605 32071 ts_tablet_manager.cc:1888] T ac1a42217d3a49d18b61b0422e164187 P 68a3c2b17ed94e06aba66ed4178132ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:07.570816 32071 log.cc:1196] T ac1a42217d3a49d18b61b0422e164187 P 68a3c2b17ed94e06aba66ed4178132ab: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal/wals/ac1a42217d3a49d18b61b0422e164187
I20240502 14:15:07.571589 26272 catalog_manager.cc:4830] TS e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): tablet ac1a42217d3a49d18b61b0422e164187 (table rebalancer_dynamic_table_new_name [id=34496715169f4212a8fd4453b575574e]) successfully deleted
I20240502 14:15:07.572094 32071 ts_tablet_manager.cc:1909] T ac1a42217d3a49d18b61b0422e164187 P 68a3c2b17ed94e06aba66ed4178132ab: Deleting consensus metadata
I20240502 14:15:07.573788 26272 catalog_manager.cc:4830] TS 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): tablet ac1a42217d3a49d18b61b0422e164187 (table rebalancer_dynamic_table_new_name [id=34496715169f4212a8fd4453b575574e]) successfully deleted
I20240502 14:15:07.634055 26278 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:15:07.645143 29554 tablet_service.cc:1467] Processing CreateTablet for tablet f236ee1fbb9545b5a76b067e67189792 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=953091122f724b7e9347c81824189960]), partition=
I20240502 14:15:07.645264 29154 tablet_service.cc:1467] Processing CreateTablet for tablet f236ee1fbb9545b5a76b067e67189792 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=953091122f724b7e9347c81824189960]), partition=
I20240502 14:15:07.645612 29554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f236ee1fbb9545b5a76b067e67189792. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:07.645670 29154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f236ee1fbb9545b5a76b067e67189792. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:07.649729 29288 tablet_service.cc:1467] Processing CreateTablet for tablet f236ee1fbb9545b5a76b067e67189792 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=953091122f724b7e9347c81824189960]), partition=
I20240502 14:15:07.650197 29288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f236ee1fbb9545b5a76b067e67189792. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:07.653430 32087 tablet_bootstrap.cc:492] T f236ee1fbb9545b5a76b067e67189792 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:15:07.655570 32087 tablet_bootstrap.cc:654] T f236ee1fbb9545b5a76b067e67189792 P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:07.667816 32087 tablet_bootstrap.cc:492] T f236ee1fbb9545b5a76b067e67189792 P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:15:07.668150 32087 ts_tablet_manager.cc:1367] T f236ee1fbb9545b5a76b067e67189792 P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.005s
I20240502 14:15:07.669390 32087 raft_consensus.cc:348] T f236ee1fbb9545b5a76b067e67189792 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:07.669843 32087 raft_consensus.cc:374] T f236ee1fbb9545b5a76b067e67189792 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:07.670056 32087 raft_consensus.cc:729] T f236ee1fbb9545b5a76b067e67189792 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:15:07.670538 32087 consensus_queue.cc:260] T f236ee1fbb9545b5a76b067e67189792 P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:07.672925 32089 tablet_bootstrap.cc:492] T f236ee1fbb9545b5a76b067e67189792 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:15:07.677197 32087 ts_tablet_manager.cc:1398] T f236ee1fbb9545b5a76b067e67189792 P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 14:15:07.680358 32089 tablet_bootstrap.cc:654] T f236ee1fbb9545b5a76b067e67189792 P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:07.683344 32091 tablet_bootstrap.cc:492] T f236ee1fbb9545b5a76b067e67189792 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:15:07.685961 32091 tablet_bootstrap.cc:654] T f236ee1fbb9545b5a76b067e67189792 P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:07.688735 32091 tablet_bootstrap.cc:492] T f236ee1fbb9545b5a76b067e67189792 P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:15:07.688985 32091 ts_tablet_manager.cc:1367] T f236ee1fbb9545b5a76b067e67189792 P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20240502 14:15:07.689895 32091 raft_consensus.cc:348] T f236ee1fbb9545b5a76b067e67189792 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:07.690217 32091 raft_consensus.cc:374] T f236ee1fbb9545b5a76b067e67189792 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:07.690371 32091 raft_consensus.cc:729] T f236ee1fbb9545b5a76b067e67189792 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:15:07.690719 32091 consensus_queue.cc:260] T f236ee1fbb9545b5a76b067e67189792 P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:07.696311 32091 ts_tablet_manager.cc:1398] T f236ee1fbb9545b5a76b067e67189792 P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:15:07.703783 32089 tablet_bootstrap.cc:492] T f236ee1fbb9545b5a76b067e67189792 P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:15:07.704044 32089 ts_tablet_manager.cc:1367] T f236ee1fbb9545b5a76b067e67189792 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.006s
I20240502 14:15:07.704980 32089 raft_consensus.cc:348] T f236ee1fbb9545b5a76b067e67189792 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:07.705250 32089 raft_consensus.cc:374] T f236ee1fbb9545b5a76b067e67189792 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:07.705374 32089 raft_consensus.cc:729] T f236ee1fbb9545b5a76b067e67189792 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:15:07.705646 32089 consensus_queue.cc:260] T f236ee1fbb9545b5a76b067e67189792 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:07.706264 32089 ts_tablet_manager.cc:1398] T f236ee1fbb9545b5a76b067e67189792 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:07.713578 32090 raft_consensus.cc:482] T f236ee1fbb9545b5a76b067e67189792 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:07.713775 32090 raft_consensus.cc:504] T f236ee1fbb9545b5a76b067e67189792 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:07.714509 32090 leader_election.cc:288] T f236ee1fbb9545b5a76b067e67189792 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:15:07.715047 29176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f236ee1fbb9545b5a76b067e67189792" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:15:07.715292 29176 raft_consensus.cc:2454] T f236ee1fbb9545b5a76b067e67189792 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:15:07.715665 29307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f236ee1fbb9545b5a76b067e67189792" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:15:07.715960 29307 raft_consensus.cc:2454] T f236ee1fbb9545b5a76b067e67189792 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:15:07.716368 29510 leader_election.cc:302] T f236ee1fbb9545b5a76b067e67189792 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 4092902b3e674857a0147f1b4f2cdb86; no voters:
I20240502 14:15:07.716661 32090 raft_consensus.cc:2781] T f236ee1fbb9545b5a76b067e67189792 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:07.716790 32090 raft_consensus.cc:482] T f236ee1fbb9545b5a76b067e67189792 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:07.716887 32090 raft_consensus.cc:3037] T f236ee1fbb9545b5a76b067e67189792 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:07.719798 32090 raft_consensus.cc:504] T f236ee1fbb9545b5a76b067e67189792 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:07.720945 32090 leader_election.cc:288] T f236ee1fbb9545b5a76b067e67189792 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Requested vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:15:07.721611 29176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f236ee1fbb9545b5a76b067e67189792" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:15:07.721668 29307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f236ee1fbb9545b5a76b067e67189792" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
I20240502 14:15:07.721814 29176 raft_consensus.cc:3037] T f236ee1fbb9545b5a76b067e67189792 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:07.721902 29307 raft_consensus.cc:3037] T f236ee1fbb9545b5a76b067e67189792 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:07.724114 29307 raft_consensus.cc:2454] T f236ee1fbb9545b5a76b067e67189792 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:15:07.724627 29176 raft_consensus.cc:2454] T f236ee1fbb9545b5a76b067e67189792 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:15:07.725407 29510 leader_election.cc:302] T f236ee1fbb9545b5a76b067e67189792 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 4092902b3e674857a0147f1b4f2cdb86; no voters:
I20240502 14:15:07.725904 32090 raft_consensus.cc:2781] T f236ee1fbb9545b5a76b067e67189792 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:07.726351 32090 raft_consensus.cc:686] T f236ee1fbb9545b5a76b067e67189792 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Becoming Leader. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Running, Role: LEADER
I20240502 14:15:07.726634 32090 consensus_queue.cc:237] T f236ee1fbb9545b5a76b067e67189792 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:07.728608 26285 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_extra_table_36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:07.730640 26278 catalog_manager.cc:5484] T f236ee1fbb9545b5a76b067e67189792 P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200). New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:15:07.751966 29288 tablet_service.cc:1467] Processing CreateTablet for tablet 65b7c6aa6aac45079adcff73f465acdb (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=123f30b9ad634d07ada549c46031d008]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:07.752516 29288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65b7c6aa6aac45079adcff73f465acdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:07.752450 29154 tablet_service.cc:1467] Processing CreateTablet for tablet 65b7c6aa6aac45079adcff73f465acdb (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=123f30b9ad634d07ada549c46031d008]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:07.752912 29154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65b7c6aa6aac45079adcff73f465acdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:07.754036 26669 tablet_service.cc:1467] Processing CreateTablet for tablet 65b7c6aa6aac45079adcff73f465acdb (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=123f30b9ad634d07ada549c46031d008]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:07.754451 26669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65b7c6aa6aac45079adcff73f465acdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:07.757251 26407 tablet_service.cc:1467] Processing CreateTablet for tablet a503853cfec1454f9a5862d1a8e5ead5 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=123f30b9ad634d07ada549c46031d008]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:07.757484 29155 tablet_service.cc:1467] Processing CreateTablet for tablet a503853cfec1454f9a5862d1a8e5ead5 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=123f30b9ad634d07ada549c46031d008]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:07.757695 26407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a503853cfec1454f9a5862d1a8e5ead5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:07.757934 29155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a503853cfec1454f9a5862d1a8e5ead5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:07.757872 29287 tablet_service.cc:1467] Processing CreateTablet for tablet f18990f33bf74c31a0faf5802cc1856d (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=123f30b9ad634d07ada549c46031d008]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:07.758288 29287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f18990f33bf74c31a0faf5802cc1856d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:07.761930 32087 tablet_bootstrap.cc:492] T 65b7c6aa6aac45079adcff73f465acdb P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:15:07.763619 32087 tablet_bootstrap.cc:654] T 65b7c6aa6aac45079adcff73f465acdb P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:07.764711 29421 tablet_service.cc:1467] Processing CreateTablet for tablet f18990f33bf74c31a0faf5802cc1856d (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=123f30b9ad634d07ada549c46031d008]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:07.764817 29422 tablet_service.cc:1467] Processing CreateTablet for tablet a503853cfec1454f9a5862d1a8e5ead5 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=123f30b9ad634d07ada549c46031d008]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:07.765169 29421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f18990f33bf74c31a0faf5802cc1856d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:07.765470 29422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a503853cfec1454f9a5862d1a8e5ead5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:07.776327 32089 tablet_bootstrap.cc:492] T f18990f33bf74c31a0faf5802cc1856d P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:15:07.777004 29554 tablet_service.cc:1467] Processing CreateTablet for tablet f18990f33bf74c31a0faf5802cc1856d (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=123f30b9ad634d07ada549c46031d008]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:07.778841 29554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f18990f33bf74c31a0faf5802cc1856d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:07.783522 32089 tablet_bootstrap.cc:654] T f18990f33bf74c31a0faf5802cc1856d P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:07.789763 32105 tablet_bootstrap.cc:492] T 65b7c6aa6aac45079adcff73f465acdb P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:15:07.795425 32105 tablet_bootstrap.cc:654] T 65b7c6aa6aac45079adcff73f465acdb P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:07.795871 32091 tablet_bootstrap.cc:492] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:15:07.799474 32091 tablet_bootstrap.cc:654] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:07.812673 32105 tablet_bootstrap.cc:492] T 65b7c6aa6aac45079adcff73f465acdb P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:15:07.812974 32105 ts_tablet_manager.cc:1367] T 65b7c6aa6aac45079adcff73f465acdb P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.003s
I20240502 14:15:07.813742 32105 raft_consensus.cc:348] T 65b7c6aa6aac45079adcff73f465acdb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:07.814100 32105 raft_consensus.cc:374] T 65b7c6aa6aac45079adcff73f465acdb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:07.814276 32105 raft_consensus.cc:729] T 65b7c6aa6aac45079adcff73f465acdb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:15:07.814646 32105 consensus_queue.cc:260] T 65b7c6aa6aac45079adcff73f465acdb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:07.815393 32104 tablet_bootstrap.cc:492] T a503853cfec1454f9a5862d1a8e5ead5 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:15:07.815527 32105 ts_tablet_manager.cc:1398] T 65b7c6aa6aac45079adcff73f465acdb P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:07.818734 32104 tablet_bootstrap.cc:654] T a503853cfec1454f9a5862d1a8e5ead5 P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:07.819447 32106 tablet_bootstrap.cc:492] T a503853cfec1454f9a5862d1a8e5ead5 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:15:07.821537 32106 tablet_bootstrap.cc:654] T a503853cfec1454f9a5862d1a8e5ead5 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:07.824434 32091 tablet_bootstrap.cc:492] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:15:07.824597 31881 raft_consensus.cc:482] T 65b7c6aa6aac45079adcff73f465acdb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:07.824756 32091 ts_tablet_manager.cc:1367] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.029s user 0.010s sys 0.002s
I20240502 14:15:07.824831 31881 raft_consensus.cc:504] T 65b7c6aa6aac45079adcff73f465acdb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:07.825615 31881 leader_election.cc:288] T 65b7c6aa6aac45079adcff73f465acdb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:15:07.825611 32091 raft_consensus.cc:348] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:07.825907 32091 raft_consensus.cc:374] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:07.825994 32091 raft_consensus.cc:729] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:15:07.826269 32091 consensus_queue.cc:260] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:07.826645 29307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b7c6aa6aac45079adcff73f465acdb" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
W20240502 14:15:07.827493 26624 leader_election.cc:341] T 65b7c6aa6aac45079adcff73f465acdb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:07.830153 29176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b7c6aa6aac45079adcff73f465acdb" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
W20240502 14:15:07.831058 26624 leader_election.cc:341] T 65b7c6aa6aac45079adcff73f465acdb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:07.831276 26624 leader_election.cc:302] T 65b7c6aa6aac45079adcff73f465acdb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters: 1cb67158133e4648a6eceba32d4dbba3, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:15:07.831965 31881 raft_consensus.cc:2726] T 65b7c6aa6aac45079adcff73f465acdb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:15:07.832502 32091 ts_tablet_manager.cc:1398] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.008s user 0.001s sys 0.002s
I20240502 14:15:07.834576 32087 tablet_bootstrap.cc:492] T 65b7c6aa6aac45079adcff73f465acdb P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:15:07.834951 32087 ts_tablet_manager.cc:1367] T 65b7c6aa6aac45079adcff73f465acdb P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.073s user 0.008s sys 0.004s
I20240502 14:15:07.836282 32087 raft_consensus.cc:348] T 65b7c6aa6aac45079adcff73f465acdb P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:07.837280 32087 raft_consensus.cc:374] T 65b7c6aa6aac45079adcff73f465acdb P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:07.837440 32087 raft_consensus.cc:729] T 65b7c6aa6aac45079adcff73f465acdb P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:15:07.837771 32087 consensus_queue.cc:260] T 65b7c6aa6aac45079adcff73f465acdb P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:07.844201 32087 ts_tablet_manager.cc:1398] T 65b7c6aa6aac45079adcff73f465acdb P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:15:07.846298 32087 tablet_bootstrap.cc:492] T a503853cfec1454f9a5862d1a8e5ead5 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:15:07.847528 32106 tablet_bootstrap.cc:492] T a503853cfec1454f9a5862d1a8e5ead5 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:15:07.847777 32106 ts_tablet_manager.cc:1367] T a503853cfec1454f9a5862d1a8e5ead5 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.000s
I20240502 14:15:07.848773 26277 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:15:07.850824 32106 raft_consensus.cc:348] T a503853cfec1454f9a5862d1a8e5ead5 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:07.852706 32087 tablet_bootstrap.cc:654] T a503853cfec1454f9a5862d1a8e5ead5 P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:07.852866 32106 raft_consensus.cc:374] T a503853cfec1454f9a5862d1a8e5ead5 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:07.853009 32106 raft_consensus.cc:729] T a503853cfec1454f9a5862d1a8e5ead5 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:15:07.853329 32106 consensus_queue.cc:260] T a503853cfec1454f9a5862d1a8e5ead5 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:07.853997 32106 ts_tablet_manager.cc:1398] T a503853cfec1454f9a5862d1a8e5ead5 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:15:07.855764 29176 raft_consensus.cc:1261] T f236ee1fbb9545b5a76b067e67189792 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:07.856406 32104 tablet_bootstrap.cc:492] T a503853cfec1454f9a5862d1a8e5ead5 P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:15:07.856685 32104 ts_tablet_manager.cc:1367] T a503853cfec1454f9a5862d1a8e5ead5 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.041s user 0.006s sys 0.000s
I20240502 14:15:07.856631 31971 consensus_queue.cc:1035] T f236ee1fbb9545b5a76b067e67189792 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:07.857688 32104 raft_consensus.cc:348] T a503853cfec1454f9a5862d1a8e5ead5 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:07.858033 32104 raft_consensus.cc:374] T a503853cfec1454f9a5862d1a8e5ead5 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:07.858196 32104 raft_consensus.cc:729] T a503853cfec1454f9a5862d1a8e5ead5 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:15:07.858490 32104 consensus_queue.cc:260] T a503853cfec1454f9a5862d1a8e5ead5 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:07.861750 32104 ts_tablet_manager.cc:1398] T a503853cfec1454f9a5862d1a8e5ead5 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:15:07.862156 32104 tablet_bootstrap.cc:492] T f18990f33bf74c31a0faf5802cc1856d P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:15:07.863705 32034 tablet.cc:1722] T f236ee1fbb9545b5a76b067e67189792 P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:07.863857 32087 tablet_bootstrap.cc:492] T a503853cfec1454f9a5862d1a8e5ead5 P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:15:07.864004 32087 ts_tablet_manager.cc:1367] T a503853cfec1454f9a5862d1a8e5ead5 P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20240502 14:15:07.864221 32104 tablet_bootstrap.cc:654] T f18990f33bf74c31a0faf5802cc1856d P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:07.864305 32034 tablet.cc:1620] T f236ee1fbb9545b5a76b067e67189792 P 4092902b3e674857a0147f1b4f2cdb86: MemRowSet was empty: no flush needed.
I20240502 14:15:07.865070 32087 raft_consensus.cc:348] T a503853cfec1454f9a5862d1a8e5ead5 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:07.865537 32087 raft_consensus.cc:374] T a503853cfec1454f9a5862d1a8e5ead5 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:07.865799 32087 raft_consensus.cc:729] T a503853cfec1454f9a5862d1a8e5ead5 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:15:07.866151 32087 consensus_queue.cc:260] T a503853cfec1454f9a5862d1a8e5ead5 P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:07.867105 32087 ts_tablet_manager.cc:1398] T a503853cfec1454f9a5862d1a8e5ead5 P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 14:15:07.878062 32104 tablet_bootstrap.cc:492] T f18990f33bf74c31a0faf5802cc1856d P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:15:07.879125 32104 ts_tablet_manager.cc:1367] T f18990f33bf74c31a0faf5802cc1856d P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.006s
I20240502 14:15:07.880442 32104 raft_consensus.cc:348] T f18990f33bf74c31a0faf5802cc1856d P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:07.880873 32104 raft_consensus.cc:374] T f18990f33bf74c31a0faf5802cc1856d P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:07.881023 32104 raft_consensus.cc:729] T f18990f33bf74c31a0faf5802cc1856d P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:15:07.881484 32104 consensus_queue.cc:260] T f18990f33bf74c31a0faf5802cc1856d P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:07.883826 32104 ts_tablet_manager.cc:1398] T f18990f33bf74c31a0faf5802cc1856d P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 14:15:07.933511 26285 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=953091122f724b7e9347c81824189960] alter complete (version 1)
I20240502 14:15:07.935322 29307 raft_consensus.cc:1261] T f236ee1fbb9545b5a76b067e67189792 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:07.937084 32089 tablet_bootstrap.cc:492] T f18990f33bf74c31a0faf5802cc1856d P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:15:07.937369 32089 ts_tablet_manager.cc:1367] T f18990f33bf74c31a0faf5802cc1856d P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.161s user 0.050s sys 0.051s
I20240502 14:15:07.940788 32117 tablet.cc:1722] T f236ee1fbb9545b5a76b067e67189792 P 1cb67158133e4648a6eceba32d4dbba3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:07.941720 32117 tablet.cc:1620] T f236ee1fbb9545b5a76b067e67189792 P 1cb67158133e4648a6eceba32d4dbba3: MemRowSet was empty: no flush needed.
I20240502 14:15:07.940001 32089 raft_consensus.cc:348] T f18990f33bf74c31a0faf5802cc1856d P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:07.943591 32089 raft_consensus.cc:374] T f18990f33bf74c31a0faf5802cc1856d P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:07.943702 32089 raft_consensus.cc:729] T f18990f33bf74c31a0faf5802cc1856d P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:15:07.944041 32089 consensus_queue.cc:260] T f18990f33bf74c31a0faf5802cc1856d P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:07.944892 31970 raft_consensus.cc:482] T a503853cfec1454f9a5862d1a8e5ead5 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:07.945050 31970 raft_consensus.cc:504] T a503853cfec1454f9a5862d1a8e5ead5 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:07.945945 31970 leader_election.cc:288] T a503853cfec1454f9a5862d1a8e5ead5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:15:07.947121 29174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a503853cfec1454f9a5862d1a8e5ead5" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:15:07.947113 29440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a503853cfec1454f9a5862d1a8e5ead5" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:15:07.947365 29174 raft_consensus.cc:2454] T a503853cfec1454f9a5862d1a8e5ead5 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:15:07.947511 29440 raft_consensus.cc:2454] T a503853cfec1454f9a5862d1a8e5ead5 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:15:07.948312 26362 leader_election.cc:302] T a503853cfec1454f9a5862d1a8e5ead5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:15:07.948628 31970 raft_consensus.cc:2781] T a503853cfec1454f9a5862d1a8e5ead5 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:07.948714 31970 raft_consensus.cc:482] T a503853cfec1454f9a5862d1a8e5ead5 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:07.948781 31970 raft_consensus.cc:3037] T a503853cfec1454f9a5862d1a8e5ead5 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:07.950690 32089 ts_tablet_manager.cc:1398] T f18990f33bf74c31a0faf5802cc1856d P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240502 14:15:07.951251 32089 tablet_bootstrap.cc:492] T 65b7c6aa6aac45079adcff73f465acdb P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:15:07.951488 31970 raft_consensus.cc:504] T a503853cfec1454f9a5862d1a8e5ead5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:07.952808 29442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a503853cfec1454f9a5862d1a8e5ead5" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
I20240502 14:15:07.953042 29442 raft_consensus.cc:3037] T a503853cfec1454f9a5862d1a8e5ead5 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:07.953364 32089 tablet_bootstrap.cc:654] T 65b7c6aa6aac45079adcff73f465acdb P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:07.955732 29442 raft_consensus.cc:2454] T a503853cfec1454f9a5862d1a8e5ead5 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:15:07.956355 26363 leader_election.cc:302] T a503853cfec1454f9a5862d1a8e5ead5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:15:07.956702 32040 raft_consensus.cc:2781] T a503853cfec1454f9a5862d1a8e5ead5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:07.957494 26285 catalog_manager.cc:5576] TS 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715) does not have the latest schema for tablet f236ee1fbb9545b5a76b067e67189792 (table rebalancer_dynamic_table [id=953091122f724b7e9347c81824189960]). Expected version 1 got 0
I20240502 14:15:07.960029 31970 leader_election.cc:288] T a503853cfec1454f9a5862d1a8e5ead5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Requested vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:15:07.963045 29176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a503853cfec1454f9a5862d1a8e5ead5" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:15:07.963353 29176 raft_consensus.cc:3037] T a503853cfec1454f9a5862d1a8e5ead5 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:07.970018 29176 raft_consensus.cc:2454] T a503853cfec1454f9a5862d1a8e5ead5 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:15:07.979921 32040 raft_consensus.cc:686] T a503853cfec1454f9a5862d1a8e5ead5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Becoming Leader. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Running, Role: LEADER
I20240502 14:15:07.980381 32040 consensus_queue.cc:237] T a503853cfec1454f9a5862d1a8e5ead5 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:07.983345 31971 consensus_queue.cc:1035] T f236ee1fbb9545b5a76b067e67189792 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:07.995838 26283 catalog_manager.cc:5484] T a503853cfec1454f9a5862d1a8e5ead5 P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: term changed from 0 to 1, leader changed from <none> to 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193). New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:08.021320 32128 tablet.cc:1722] T f236ee1fbb9545b5a76b067e67189792 P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:08.025228 32128 tablet.cc:1620] T f236ee1fbb9545b5a76b067e67189792 P c1100e16b3b9495ca61d4f95e89b6a89: MemRowSet was empty: no flush needed.
I20240502 14:15:08.034847 26283 catalog_manager.cc:5576] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659) does not have the latest schema for tablet f236ee1fbb9545b5a76b067e67189792 (table rebalancer_dynamic_table [id=953091122f724b7e9347c81824189960]). Expected version 1 got 0
I20240502 14:15:08.037776 32089 tablet_bootstrap.cc:492] T 65b7c6aa6aac45079adcff73f465acdb P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:15:08.037961 32089 ts_tablet_manager.cc:1367] T 65b7c6aa6aac45079adcff73f465acdb P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.087s user 0.021s sys 0.044s
I20240502 14:15:08.038908 32089 raft_consensus.cc:348] T 65b7c6aa6aac45079adcff73f465acdb P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:08.039129 32089 raft_consensus.cc:374] T 65b7c6aa6aac45079adcff73f465acdb P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:08.039223 32089 raft_consensus.cc:729] T 65b7c6aa6aac45079adcff73f465acdb P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:15:08.039553 32089 consensus_queue.cc:260] T 65b7c6aa6aac45079adcff73f465acdb P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:08.042400 32089 ts_tablet_manager.cc:1398] T 65b7c6aa6aac45079adcff73f465acdb P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:15:08.048348 32119 raft_consensus.cc:482] T f18990f33bf74c31a0faf5802cc1856d P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:08.048712 32119 raft_consensus.cc:504] T f18990f33bf74c31a0faf5802cc1856d P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:08.050209 29442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f18990f33bf74c31a0faf5802cc1856d" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:15:08.050469 29442 raft_consensus.cc:2454] T f18990f33bf74c31a0faf5802cc1856d P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 0.
I20240502 14:15:08.051136 29245 leader_election.cc:302] T f18990f33bf74c31a0faf5802cc1856d P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:15:08.051559 31946 raft_consensus.cc:2781] T f18990f33bf74c31a0faf5802cc1856d P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:08.051692 31946 raft_consensus.cc:482] T f18990f33bf74c31a0faf5802cc1856d P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:08.051785 31946 raft_consensus.cc:3037] T f18990f33bf74c31a0faf5802cc1856d P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:08.054970 31946 raft_consensus.cc:504] T f18990f33bf74c31a0faf5802cc1856d P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:08.060966 26285 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:15:08.072857 29442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f18990f33bf74c31a0faf5802cc1856d" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
I20240502 14:15:08.073184 29442 raft_consensus.cc:3037] T f18990f33bf74c31a0faf5802cc1856d P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:08.086321 31946 leader_election.cc:288] T f18990f33bf74c31a0faf5802cc1856d P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 election: Requested vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:15:08.087010 32034 tablet.cc:1722] T f236ee1fbb9545b5a76b067e67189792 P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:08.087780 32034 tablet.cc:1620] T f236ee1fbb9545b5a76b067e67189792 P 4092902b3e674857a0147f1b4f2cdb86: MemRowSet was empty: no flush needed.
I20240502 14:15:08.088498 29442 raft_consensus.cc:2454] T f18990f33bf74c31a0faf5802cc1856d P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 1.
I20240502 14:15:08.090831 32119 leader_election.cc:288] T f18990f33bf74c31a0faf5802cc1856d P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:15:08.093420 32117 tablet.cc:1722] T f236ee1fbb9545b5a76b067e67189792 P 1cb67158133e4648a6eceba32d4dbba3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:08.094017 32117 tablet.cc:1620] T f236ee1fbb9545b5a76b067e67189792 P 1cb67158133e4648a6eceba32d4dbba3: MemRowSet was empty: no flush needed.
I20240502 14:15:08.097515 29245 leader_election.cc:302] T f18990f33bf74c31a0faf5802cc1856d P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:15:08.097898 32133 raft_consensus.cc:2781] T f18990f33bf74c31a0faf5802cc1856d P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:08.098492 32133 raft_consensus.cc:686] T f18990f33bf74c31a0faf5802cc1856d P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 LEADER]: Becoming Leader. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Running, Role: LEADER
I20240502 14:15:08.099448 29575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f18990f33bf74c31a0faf5802cc1856d" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86" is_pre_election: true
I20240502 14:15:08.099540 32133 consensus_queue.cc:237] T f18990f33bf74c31a0faf5802cc1856d P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:08.099754 29575 raft_consensus.cc:2454] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 0.
I20240502 14:15:08.100570 29574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f18990f33bf74c31a0faf5802cc1856d" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86"
I20240502 14:15:08.100775 29574 raft_consensus.cc:3037] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:08.112505 26285 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=953091122f724b7e9347c81824189960] alter complete (version 2)
I20240502 14:15:08.138057 29574 raft_consensus.cc:2454] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 1.
I20240502 14:15:08.140975 26278 catalog_manager.cc:5484] T f18990f33bf74c31a0faf5802cc1856d P c1100e16b3b9495ca61d4f95e89b6a89 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198). New cstate: current_term: 1 leader_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:15:08.146054 32128 tablet.cc:1722] T f236ee1fbb9545b5a76b067e67189792 P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:08.147050 32128 tablet.cc:1620] T f236ee1fbb9545b5a76b067e67189792 P c1100e16b3b9495ca61d4f95e89b6a89: MemRowSet was empty: no flush needed.
I20240502 14:15:08.268643 26276 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:15:08.292301 32034 tablet.cc:1722] T f236ee1fbb9545b5a76b067e67189792 P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:08.294852 32117 tablet.cc:1722] T f236ee1fbb9545b5a76b067e67189792 P 1cb67158133e4648a6eceba32d4dbba3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:08.307140 26284 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table_new_name [id=953091122f724b7e9347c81824189960] alter complete (version 3)
I20240502 14:15:08.309692 32128 tablet.cc:1722] T f236ee1fbb9545b5a76b067e67189792 P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:08.312566 31917 raft_consensus.cc:482] T 65b7c6aa6aac45079adcff73f465acdb P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:08.312798 31917 raft_consensus.cc:504] T 65b7c6aa6aac45079adcff73f465acdb P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:08.313488 31917 leader_election.cc:288] T 65b7c6aa6aac45079adcff73f465acdb P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:15:08.314297 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b7c6aa6aac45079adcff73f465acdb" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:15:08.314566 29309 raft_consensus.cc:2454] T 65b7c6aa6aac45079adcff73f465acdb P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 0.
I20240502 14:15:08.315104 29111 leader_election.cc:302] T 65b7c6aa6aac45079adcff73f465acdb P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:15:08.316267 26689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b7c6aa6aac45079adcff73f465acdb" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:15:08.316545 26689 raft_consensus.cc:2454] T 65b7c6aa6aac45079adcff73f465acdb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 0.
I20240502 14:15:08.324796 31917 raft_consensus.cc:2781] T 65b7c6aa6aac45079adcff73f465acdb P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:08.325085 31917 raft_consensus.cc:482] T 65b7c6aa6aac45079adcff73f465acdb P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:08.325287 31917 raft_consensus.cc:3037] T 65b7c6aa6aac45079adcff73f465acdb P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:08.341125 31917 raft_consensus.cc:504] T 65b7c6aa6aac45079adcff73f465acdb P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:08.345290 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b7c6aa6aac45079adcff73f465acdb" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
I20240502 14:15:08.345585 29309 raft_consensus.cc:3037] T 65b7c6aa6aac45079adcff73f465acdb P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:08.353202 29309 raft_consensus.cc:2454] T 65b7c6aa6aac45079adcff73f465acdb P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 1.
I20240502 14:15:08.353937 29111 leader_election.cc:302] T 65b7c6aa6aac45079adcff73f465acdb P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:15:08.354429 32069 raft_consensus.cc:2781] T 65b7c6aa6aac45079adcff73f465acdb P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:08.358963 32040 consensus_queue.cc:1035] T a503853cfec1454f9a5862d1a8e5ead5 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:08.370190 26689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b7c6aa6aac45079adcff73f465acdb" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:15:08.370524 26689 raft_consensus.cc:3037] T 65b7c6aa6aac45079adcff73f465acdb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:08.374490 26689 raft_consensus.cc:2454] T 65b7c6aa6aac45079adcff73f465acdb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 1.
I20240502 14:15:08.377379 31917 leader_election.cc:288] T 65b7c6aa6aac45079adcff73f465acdb P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 election: Requested vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:15:08.388350 32069 raft_consensus.cc:686] T 65b7c6aa6aac45079adcff73f465acdb P 1cb67158133e4648a6eceba32d4dbba3 [term 1 LEADER]: Becoming Leader. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Running, Role: LEADER
I20240502 14:15:08.388926 32069 consensus_queue.cc:237] T 65b7c6aa6aac45079adcff73f465acdb P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:08.404150 26277 catalog_manager.cc:5484] T 65b7c6aa6aac45079adcff73f465acdb P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197). New cstate: current_term: 1 leader_uuid: "1cb67158133e4648a6eceba32d4dbba3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:08.413723 32123 consensus_queue.cc:1035] T a503853cfec1454f9a5862d1a8e5ead5 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:08.480706 26277 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:08.481040 26277 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:08.515991 26277 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet f236ee1fbb9545b5a76b067e67189792
I20240502 14:15:08.517199 29155 tablet_service.cc:1514] Processing DeleteTablet for tablet f236ee1fbb9545b5a76b067e67189792 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:08 UTC) from {username='slave'} at 127.0.0.1:55202
I20240502 14:15:08.517987 32169 tablet_replica.cc:330] T f236ee1fbb9545b5a76b067e67189792 P 1cb67158133e4648a6eceba32d4dbba3: stopping tablet replica
I20240502 14:15:08.519740 32169 raft_consensus.cc:2229] T f236ee1fbb9545b5a76b067e67189792 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:08.521113 32169 raft_consensus.cc:2258] T f236ee1fbb9545b5a76b067e67189792 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:08.522410 32169 ts_tablet_manager.cc:1875] T f236ee1fbb9545b5a76b067e67189792 P 1cb67158133e4648a6eceba32d4dbba3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:08.522801 29554 tablet_service.cc:1514] Processing DeleteTablet for tablet f236ee1fbb9545b5a76b067e67189792 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:08 UTC) from {username='slave'} at 127.0.0.1:36398
I20240502 14:15:08.523531 32170 tablet_replica.cc:330] T f236ee1fbb9545b5a76b067e67189792 P 4092902b3e674857a0147f1b4f2cdb86: stopping tablet replica
I20240502 14:15:08.523838 32170 raft_consensus.cc:2229] T f236ee1fbb9545b5a76b067e67189792 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:15:08.524336 32170 raft_consensus.cc:2258] T f236ee1fbb9545b5a76b067e67189792 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:08.526505 29288 tablet_service.cc:1514] Processing DeleteTablet for tablet f236ee1fbb9545b5a76b067e67189792 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:08 UTC) from {username='slave'} at 127.0.0.1:59160
I20240502 14:15:08.530871 32170 ts_tablet_manager.cc:1875] T f236ee1fbb9545b5a76b067e67189792 P 4092902b3e674857a0147f1b4f2cdb86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:08.544652 32171 tablet_replica.cc:330] T f236ee1fbb9545b5a76b067e67189792 P c1100e16b3b9495ca61d4f95e89b6a89: stopping tablet replica
I20240502 14:15:08.555302 32171 raft_consensus.cc:2229] T f236ee1fbb9545b5a76b067e67189792 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:08.555698 32171 raft_consensus.cc:2258] T f236ee1fbb9545b5a76b067e67189792 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:08.556857 32171 ts_tablet_manager.cc:1875] T f236ee1fbb9545b5a76b067e67189792 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:08.563942 32171 ts_tablet_manager.cc:1888] T f236ee1fbb9545b5a76b067e67189792 P c1100e16b3b9495ca61d4f95e89b6a89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:08.564210 32171 log.cc:1196] T f236ee1fbb9545b5a76b067e67189792 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal/wals/f236ee1fbb9545b5a76b067e67189792
I20240502 14:15:08.564765 32171 ts_tablet_manager.cc:1909] T f236ee1fbb9545b5a76b067e67189792 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting consensus metadata
I20240502 14:15:08.566082 26271 catalog_manager.cc:4830] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): tablet f236ee1fbb9545b5a76b067e67189792 (table rebalancer_dynamic_table_new_name [id=953091122f724b7e9347c81824189960]) successfully deleted
I20240502 14:15:08.584437 32169 ts_tablet_manager.cc:1888] T f236ee1fbb9545b5a76b067e67189792 P 1cb67158133e4648a6eceba32d4dbba3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:08.585670 32169 log.cc:1196] T f236ee1fbb9545b5a76b067e67189792 P 1cb67158133e4648a6eceba32d4dbba3: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal/wals/f236ee1fbb9545b5a76b067e67189792
I20240502 14:15:08.586666 32170 ts_tablet_manager.cc:1888] T f236ee1fbb9545b5a76b067e67189792 P 4092902b3e674857a0147f1b4f2cdb86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:08.587325 32169 ts_tablet_manager.cc:1909] T f236ee1fbb9545b5a76b067e67189792 P 1cb67158133e4648a6eceba32d4dbba3: Deleting consensus metadata
I20240502 14:15:08.589390 32170 log.cc:1196] T f236ee1fbb9545b5a76b067e67189792 P 4092902b3e674857a0147f1b4f2cdb86: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/wal/wals/f236ee1fbb9545b5a76b067e67189792
I20240502 14:15:08.589390 26271 catalog_manager.cc:4830] TS 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): tablet f236ee1fbb9545b5a76b067e67189792 (table rebalancer_dynamic_table_new_name [id=953091122f724b7e9347c81824189960]) successfully deleted
I20240502 14:15:08.590888 32170 ts_tablet_manager.cc:1909] T f236ee1fbb9545b5a76b067e67189792 P 4092902b3e674857a0147f1b4f2cdb86: Deleting consensus metadata
I20240502 14:15:08.597976 26272 catalog_manager.cc:4830] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097): tablet f236ee1fbb9545b5a76b067e67189792 (table rebalancer_dynamic_table_new_name [id=953091122f724b7e9347c81824189960]) successfully deleted
I20240502 14:15:08.629956 26277 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:15:08.640343 32119 consensus_queue.cc:1035] T f18990f33bf74c31a0faf5802cc1856d P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:08.652652 32133 consensus_queue.cc:1035] T f18990f33bf74c31a0faf5802cc1856d P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:08.693882 29554 tablet_service.cc:1467] Processing CreateTablet for tablet ba860bfc9ad941ac994e336df805d404 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c7b6f3bedc784c2eb7a081c8626ec47d]), partition=
I20240502 14:15:08.694639 29554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba860bfc9ad941ac994e336df805d404. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:08.694603 26797 tablet_service.cc:1467] Processing CreateTablet for tablet ba860bfc9ad941ac994e336df805d404 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c7b6f3bedc784c2eb7a081c8626ec47d]), partition=
I20240502 14:15:08.694981 26797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba860bfc9ad941ac994e336df805d404. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:08.755702 32183 tablet_bootstrap.cc:492] T ba860bfc9ad941ac994e336df805d404 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:15:08.770272 32183 tablet_bootstrap.cc:654] T ba860bfc9ad941ac994e336df805d404 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:08.782054 29155 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:08.789327 32187 tablet_bootstrap.cc:492] T ba860bfc9ad941ac994e336df805d404 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:15:08.790022 26669 tablet_service.cc:1467] Processing CreateTablet for tablet ba860bfc9ad941ac994e336df805d404 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c7b6f3bedc784c2eb7a081c8626ec47d]), partition=
I20240502 14:15:08.790467 26669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba860bfc9ad941ac994e336df805d404. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:08.847219 32191 tablet_bootstrap.cc:492] T ba860bfc9ad941ac994e336df805d404 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:15:08.852759 32187 tablet_bootstrap.cc:654] T ba860bfc9ad941ac994e336df805d404 P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:08.855247 32191 tablet_bootstrap.cc:654] T ba860bfc9ad941ac994e336df805d404 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:08.860556 32183 tablet_bootstrap.cc:492] T ba860bfc9ad941ac994e336df805d404 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:15:08.861666 32187 tablet_bootstrap.cc:492] T ba860bfc9ad941ac994e336df805d404 P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:15:08.861917 32187 ts_tablet_manager.cc:1367] T ba860bfc9ad941ac994e336df805d404 P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.073s user 0.006s sys 0.000s
I20240502 14:15:08.862015 32183 ts_tablet_manager.cc:1367] T ba860bfc9ad941ac994e336df805d404 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.106s user 0.007s sys 0.013s
I20240502 14:15:08.862959 32187 raft_consensus.cc:348] T ba860bfc9ad941ac994e336df805d404 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:08.862951 32183 raft_consensus.cc:348] T ba860bfc9ad941ac994e336df805d404 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:08.863197 32183 raft_consensus.cc:374] T ba860bfc9ad941ac994e336df805d404 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:08.863353 32187 raft_consensus.cc:374] T ba860bfc9ad941ac994e336df805d404 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:08.863831 32187 raft_consensus.cc:729] T ba860bfc9ad941ac994e336df805d404 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:15:08.864418 32187 consensus_queue.cc:260] T ba860bfc9ad941ac994e336df805d404 P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:08.868032 32183 raft_consensus.cc:729] T ba860bfc9ad941ac994e336df805d404 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:15:08.870473 32187 ts_tablet_manager.cc:1398] T ba860bfc9ad941ac994e336df805d404 P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240502 14:15:08.870299 32183 consensus_queue.cc:260] T ba860bfc9ad941ac994e336df805d404 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:08.872749 32183 ts_tablet_manager.cc:1398] T ba860bfc9ad941ac994e336df805d404 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 14:15:08.873828 32069 consensus_queue.cc:1035] T 65b7c6aa6aac45079adcff73f465acdb P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:08.887387 32191 tablet_bootstrap.cc:492] T ba860bfc9ad941ac994e336df805d404 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:15:08.888157 32191 ts_tablet_manager.cc:1367] T ba860bfc9ad941ac994e336df805d404 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.000s
I20240502 14:15:08.889214 32191 raft_consensus.cc:348] T ba860bfc9ad941ac994e336df805d404 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:08.889562 32191 raft_consensus.cc:374] T ba860bfc9ad941ac994e336df805d404 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:08.889750 32191 raft_consensus.cc:729] T ba860bfc9ad941ac994e336df805d404 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:15:08.890722 32157 consensus_queue.cc:1035] T 65b7c6aa6aac45079adcff73f465acdb P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:08.891026 32191 consensus_queue.cc:260] T ba860bfc9ad941ac994e336df805d404 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:08.906821 32191 ts_tablet_manager.cc:1398] T ba860bfc9ad941ac994e336df805d404 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.018s user 0.000s sys 0.002s
I20240502 14:15:08.957090 29422 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:15:08.957562 29554 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:15:08.988435 32179 raft_consensus.cc:482] T ba860bfc9ad941ac994e336df805d404 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:08.988638 32179 raft_consensus.cc:504] T ba860bfc9ad941ac994e336df805d404 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:08.989894 32179 leader_election.cc:288] T ba860bfc9ad941ac994e336df805d404 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:15:08.991405 26819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba860bfc9ad941ac994e336df805d404" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:15:08.991820 26819 raft_consensus.cc:2454] T ba860bfc9ad941ac994e336df805d404 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:15:08.992347 26625 leader_election.cc:302] T ba860bfc9ad941ac994e336df805d404 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:15:08.992283 29574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba860bfc9ad941ac994e336df805d404" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86" is_pre_election: true
I20240502 14:15:08.992631 29574 raft_consensus.cc:2454] T ba860bfc9ad941ac994e336df805d404 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:15:08.994477 32179 raft_consensus.cc:2781] T ba860bfc9ad941ac994e336df805d404 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:08.994630 32179 raft_consensus.cc:482] T ba860bfc9ad941ac994e336df805d404 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:08.994722 32179 raft_consensus.cc:3037] T ba860bfc9ad941ac994e336df805d404 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:08.998229 32179 raft_consensus.cc:504] T ba860bfc9ad941ac994e336df805d404 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:08.999171 32179 leader_election.cc:288] T ba860bfc9ad941ac994e336df805d404 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Requested vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:15:09.002442 29574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba860bfc9ad941ac994e336df805d404" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86"
I20240502 14:15:09.003418 29574 raft_consensus.cc:3037] T ba860bfc9ad941ac994e336df805d404 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:09.004616 26277 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_extra_table_38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:09.004956 26819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba860bfc9ad941ac994e336df805d404" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:15:09.005215 26819 raft_consensus.cc:3037] T ba860bfc9ad941ac994e336df805d404 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:09.009168 29574 raft_consensus.cc:2454] T ba860bfc9ad941ac994e336df805d404 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:15:09.014452 26625 leader_election.cc:302] T ba860bfc9ad941ac994e336df805d404 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters:
I20240502 14:15:09.015187 32178 raft_consensus.cc:2781] T ba860bfc9ad941ac994e336df805d404 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:09.015350 32178 raft_consensus.cc:686] T ba860bfc9ad941ac994e336df805d404 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Becoming Leader. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Running, Role: LEADER
I20240502 14:15:09.015697 32178 consensus_queue.cc:237] T ba860bfc9ad941ac994e336df805d404 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:09.021430 29554 tablet_service.cc:1467] Processing CreateTablet for tablet 88b726d7142846d6893f4f3b763c0565 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=7a3bd56fd0ef4a5f89e69991dd8d785a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:09.021992 29554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88b726d7142846d6893f4f3b763c0565. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:09.024348 26538 tablet_service.cc:1467] Processing CreateTablet for tablet ebbe2bc571aa44d0a093819144ce49b1 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=7a3bd56fd0ef4a5f89e69991dd8d785a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:09.024868 26538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebbe2bc571aa44d0a093819144ce49b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:09.028940 29155 tablet_service.cc:1467] Processing CreateTablet for tablet ebbe2bc571aa44d0a093819144ce49b1 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=7a3bd56fd0ef4a5f89e69991dd8d785a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:09.029500 29155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebbe2bc571aa44d0a093819144ce49b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:09.033769 29288 tablet_service.cc:1467] Processing CreateTablet for tablet ebbe2bc571aa44d0a093819144ce49b1 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=7a3bd56fd0ef4a5f89e69991dd8d785a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:09.034310 29288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebbe2bc571aa44d0a093819144ce49b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:09.038805 29553 tablet_service.cc:1467] Processing CreateTablet for tablet 23f576e1bede478496122a9d1a2e7515 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=7a3bd56fd0ef4a5f89e69991dd8d785a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:09.039333 29553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23f576e1bede478496122a9d1a2e7515. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:09.036952 29287 tablet_service.cc:1467] Processing CreateTablet for tablet 23f576e1bede478496122a9d1a2e7515 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=7a3bd56fd0ef4a5f89e69991dd8d785a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:09.039709 29422 tablet_service.cc:1467] Processing CreateTablet for tablet 88b726d7142846d6893f4f3b763c0565 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=7a3bd56fd0ef4a5f89e69991dd8d785a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:09.040063 29287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23f576e1bede478496122a9d1a2e7515. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:09.040318 29422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88b726d7142846d6893f4f3b763c0565. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:09.041196 26797 tablet_service.cc:1467] Processing CreateTablet for tablet 88b726d7142846d6893f4f3b763c0565 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=7a3bd56fd0ef4a5f89e69991dd8d785a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:09.041659 26797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88b726d7142846d6893f4f3b763c0565. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:09.045224 29421 tablet_service.cc:1467] Processing CreateTablet for tablet 23f576e1bede478496122a9d1a2e7515 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=7a3bd56fd0ef4a5f89e69991dd8d785a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:09.045683 29421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23f576e1bede478496122a9d1a2e7515. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:09.051491 32187 tablet_bootstrap.cc:492] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:15:09.053619 32187 tablet_bootstrap.cc:654] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:09.062737 32212 tablet_bootstrap.cc:492] T 23f576e1bede478496122a9d1a2e7515 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:15:09.063298 26277 catalog_manager.cc:5484] T ba860bfc9ad941ac994e336df805d404 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195). New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:09.064723 32212 tablet_bootstrap.cc:654] T 23f576e1bede478496122a9d1a2e7515 P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:09.083424 26819 raft_consensus.cc:2454] T ba860bfc9ad941ac994e336df805d404 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:15:09.089902 32183 tablet_bootstrap.cc:492] T 88b726d7142846d6893f4f3b763c0565 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:15:09.091830 32183 tablet_bootstrap.cc:654] T 88b726d7142846d6893f4f3b763c0565 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:09.092716 32212 tablet_bootstrap.cc:492] T 23f576e1bede478496122a9d1a2e7515 P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:15:09.092989 32212 ts_tablet_manager.cc:1367] T 23f576e1bede478496122a9d1a2e7515 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.030s user 0.010s sys 0.000s
I20240502 14:15:09.093868 32212 raft_consensus.cc:348] T 23f576e1bede478496122a9d1a2e7515 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:09.094177 32212 raft_consensus.cc:374] T 23f576e1bede478496122a9d1a2e7515 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:09.094313 32212 raft_consensus.cc:729] T 23f576e1bede478496122a9d1a2e7515 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:15:09.094676 32212 consensus_queue.cc:260] T 23f576e1bede478496122a9d1a2e7515 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:09.101300 32212 ts_tablet_manager.cc:1398] T 23f576e1bede478496122a9d1a2e7515 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 14:15:09.104210 32212 tablet_bootstrap.cc:492] T 88b726d7142846d6893f4f3b763c0565 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:15:09.106395 32212 tablet_bootstrap.cc:654] T 88b726d7142846d6893f4f3b763c0565 P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:09.122241 32216 tablet_bootstrap.cc:492] T 23f576e1bede478496122a9d1a2e7515 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:15:09.130422 26537 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:09.129487 32215 tablet_bootstrap.cc:492] T ebbe2bc571aa44d0a093819144ce49b1 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:15:09.133622 26407 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:15:09.140923 32183 tablet_bootstrap.cc:492] T 88b726d7142846d6893f4f3b763c0565 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:15:09.141255 32183 ts_tablet_manager.cc:1367] T 88b726d7142846d6893f4f3b763c0565 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.051s user 0.010s sys 0.000s
I20240502 14:15:09.140584 32187 tablet_bootstrap.cc:492] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:15:09.142995 32187 ts_tablet_manager.cc:1367] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.092s user 0.003s sys 0.006s
I20240502 14:15:09.142815 32183 raft_consensus.cc:348] T 88b726d7142846d6893f4f3b763c0565 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:09.143415 32183 raft_consensus.cc:374] T 88b726d7142846d6893f4f3b763c0565 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:09.143565 32183 raft_consensus.cc:729] T 88b726d7142846d6893f4f3b763c0565 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:15:09.143937 32183 consensus_queue.cc:260] T 88b726d7142846d6893f4f3b763c0565 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:09.144040 32187 raft_consensus.cc:348] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:09.144557 32187 raft_consensus.cc:374] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:09.144651 32187 raft_consensus.cc:729] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:15:09.144928 32187 consensus_queue.cc:260] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:09.149852 32183 ts_tablet_manager.cc:1398] T 88b726d7142846d6893f4f3b763c0565 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 14:15:09.149739 31623 raft_consensus.cc:482] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:09.152551 31623 raft_consensus.cc:504] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:09.153548 31623 leader_election.cc:288] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:15:09.154276 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f576e1bede478496122a9d1a2e7515" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:15:09.154532 29443 raft_consensus.cc:2454] T 23f576e1bede478496122a9d1a2e7515 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:15:09.155061 29511 leader_election.cc:302] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, 68a3c2b17ed94e06aba66ed4178132ab; no voters:
I20240502 14:15:09.155416 31623 raft_consensus.cc:2781] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:09.155560 31623 raft_consensus.cc:482] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:09.155680 31623 raft_consensus.cc:3037] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:09.156098 32215 tablet_bootstrap.cc:654] T ebbe2bc571aa44d0a093819144ce49b1 P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:09.156471 32216 tablet_bootstrap.cc:654] T 23f576e1bede478496122a9d1a2e7515 P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:09.157797 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f576e1bede478496122a9d1a2e7515" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
W20240502 14:15:09.158988 29510 leader_election.cc:341] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:09.159461 31623 raft_consensus.cc:504] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:09.160761 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f576e1bede478496122a9d1a2e7515" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
W20240502 14:15:09.161866 29510 leader_election.cc:341] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:09.162150 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f576e1bede478496122a9d1a2e7515" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
I20240502 14:15:09.162441 29443 raft_consensus.cc:3037] T 23f576e1bede478496122a9d1a2e7515 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:09.165477 32187 ts_tablet_manager.cc:1398] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240502 14:15:09.167733 29443 raft_consensus.cc:2454] T 23f576e1bede478496122a9d1a2e7515 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:15:09.171207 29511 leader_election.cc:302] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, 68a3c2b17ed94e06aba66ed4178132ab; no voters: c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:15:09.171624 31971 raft_consensus.cc:2781] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:09.171798 31971 raft_consensus.cc:686] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Becoming Leader. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Running, Role: LEADER
I20240502 14:15:09.172117 31971 consensus_queue.cc:237] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:09.173208 32215 tablet_bootstrap.cc:492] T ebbe2bc571aa44d0a093819144ce49b1 P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:15:09.173897 32215 ts_tablet_manager.cc:1367] T ebbe2bc571aa44d0a093819144ce49b1 P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.045s user 0.004s sys 0.002s
I20240502 14:15:09.174077 26276 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:15:09.176834 32216 tablet_bootstrap.cc:492] T 23f576e1bede478496122a9d1a2e7515 P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:15:09.179342 32216 ts_tablet_manager.cc:1367] T 23f576e1bede478496122a9d1a2e7515 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.057s user 0.004s sys 0.003s
I20240502 14:15:09.183338 29574 raft_consensus.cc:1261] T ba860bfc9ad941ac994e336df805d404 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:09.184104 31617 consensus_queue.cc:1035] T ba860bfc9ad941ac994e336df805d404 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:09.185248 31623 leader_election.cc:288] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Requested vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:15:09.187661 26819 raft_consensus.cc:1261] T ba860bfc9ad941ac994e336df805d404 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:09.188588 31617 consensus_queue.cc:1035] T ba860bfc9ad941ac994e336df805d404 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:09.193090 32187 tablet_bootstrap.cc:492] T 88b726d7142846d6893f4f3b763c0565 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:15:09.197289 29287 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:15:09.204797 26669 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:15:09.207393 32187 tablet_bootstrap.cc:654] T 88b726d7142846d6893f4f3b763c0565 P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:09.209210 26797 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:09.227078 26277 catalog_manager.cc:5484] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200). New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:15:09.228849 32138 raft_consensus.cc:482] T 88b726d7142846d6893f4f3b763c0565 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:09.229050 32138 raft_consensus.cc:504] T 88b726d7142846d6893f4f3b763c0565 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:09.230222 32138 leader_election.cc:288] T 88b726d7142846d6893f4f3b763c0565 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:15:09.230594 29575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88b726d7142846d6893f4f3b763c0565" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86" is_pre_election: true
W20240502 14:15:09.231343 26755 leader_election.cc:341] T 88b726d7142846d6893f4f3b763c0565 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:09.231948 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88b726d7142846d6893f4f3b763c0565" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
W20240502 14:15:09.232642 26755 leader_election.cc:341] T 88b726d7142846d6893f4f3b763c0565 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:09.232767 26755 leader_election.cc:302] T 88b726d7142846d6893f4f3b763c0565 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4cfb8fc5a474177af6848a6efa13a27; no voters: 4092902b3e674857a0147f1b4f2cdb86, 68a3c2b17ed94e06aba66ed4178132ab
I20240502 14:15:09.233291 32138 raft_consensus.cc:2726] T 88b726d7142846d6893f4f3b763c0565 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:15:09.182487 32215 raft_consensus.cc:348] T ebbe2bc571aa44d0a093819144ce49b1 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:09.253816 32187 tablet_bootstrap.cc:492] T 88b726d7142846d6893f4f3b763c0565 P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:15:09.254045 32187 ts_tablet_manager.cc:1367] T 88b726d7142846d6893f4f3b763c0565 P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.061s user 0.004s sys 0.000s
I20240502 14:15:09.255836 32187 raft_consensus.cc:348] T 88b726d7142846d6893f4f3b763c0565 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:09.256073 32187 raft_consensus.cc:374] T 88b726d7142846d6893f4f3b763c0565 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:09.256165 32187 raft_consensus.cc:729] T 88b726d7142846d6893f4f3b763c0565 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:15:09.260705 32187 consensus_queue.cc:260] T 88b726d7142846d6893f4f3b763c0565 P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:09.261046 32211 tablet_bootstrap.cc:492] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:15:09.262562 32212 tablet_bootstrap.cc:492] T 88b726d7142846d6893f4f3b763c0565 P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:15:09.262815 32212 ts_tablet_manager.cc:1367] T 88b726d7142846d6893f4f3b763c0565 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.159s user 0.060s sys 0.034s
I20240502 14:15:09.263672 32212 raft_consensus.cc:348] T 88b726d7142846d6893f4f3b763c0565 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:09.263994 32212 raft_consensus.cc:374] T 88b726d7142846d6893f4f3b763c0565 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:09.264127 32212 raft_consensus.cc:729] T 88b726d7142846d6893f4f3b763c0565 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:15:09.264478 32212 consensus_queue.cc:260] T 88b726d7142846d6893f4f3b763c0565 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:09.266101 32187 ts_tablet_manager.cc:1398] T 88b726d7142846d6893f4f3b763c0565 P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.012s user 0.002s sys 0.001s
I20240502 14:15:09.266391 32212 ts_tablet_manager.cc:1398] T 88b726d7142846d6893f4f3b763c0565 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:15:09.280179 32211 tablet_bootstrap.cc:654] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:09.281884 32227 tablet.cc:1722] T ba860bfc9ad941ac994e336df805d404 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:09.282897 32227 tablet.cc:1620] T ba860bfc9ad941ac994e336df805d404 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: MemRowSet was empty: no flush needed.
I20240502 14:15:09.289949 32215 raft_consensus.cc:374] T ebbe2bc571aa44d0a093819144ce49b1 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:09.290108 32215 raft_consensus.cc:729] T ebbe2bc571aa44d0a093819144ce49b1 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:15:09.290366 32215 consensus_queue.cc:260] T ebbe2bc571aa44d0a093819144ce49b1 P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:09.291021 32215 ts_tablet_manager.cc:1398] T ebbe2bc571aa44d0a093819144ce49b1 P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.117s user 0.000s sys 0.001s
I20240502 14:15:09.293522 26277 catalog_manager.cc:5576] TS e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843) does not have the latest schema for tablet ba860bfc9ad941ac994e336df805d404 (table rebalancer_dynamic_table [id=c7b6f3bedc784c2eb7a081c8626ec47d]). Expected version 1 got 0
I20240502 14:15:09.293743 32216 raft_consensus.cc:348] T 23f576e1bede478496122a9d1a2e7515 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:09.294034 32216 raft_consensus.cc:374] T 23f576e1bede478496122a9d1a2e7515 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:09.294559 32216 raft_consensus.cc:729] T 23f576e1bede478496122a9d1a2e7515 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:15:09.296137 32216 consensus_queue.cc:260] T 23f576e1bede478496122a9d1a2e7515 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:09.296967 32216 ts_tablet_manager.cc:1398] T 23f576e1bede478496122a9d1a2e7515 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.116s user 0.002s sys 0.000s
I20240502 14:15:09.299435 32216 tablet_bootstrap.cc:492] T ebbe2bc571aa44d0a093819144ce49b1 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:15:09.325138 32211 tablet_bootstrap.cc:492] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:15:09.327181 32211 ts_tablet_manager.cc:1367] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.068s user 0.007s sys 0.000s
I20240502 14:15:09.330787 32229 tablet.cc:1722] T ba860bfc9ad941ac994e336df805d404 P e4cfb8fc5a474177af6848a6efa13a27: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:09.331843 32211 raft_consensus.cc:348] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:09.332155 32211 raft_consensus.cc:374] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:09.333370 32211 raft_consensus.cc:729] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:15:09.334069 32211 consensus_queue.cc:260] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:09.336315 32211 ts_tablet_manager.cc:1398] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 14:15:09.344218 26277 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=c7b6f3bedc784c2eb7a081c8626ec47d] alter complete (version 1)
I20240502 14:15:09.351467 26283 catalog_manager.cc:5576] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097) does not have the latest schema for tablet ba860bfc9ad941ac994e336df805d404 (table rebalancer_dynamic_table [id=c7b6f3bedc784c2eb7a081c8626ec47d]). Expected version 1 got 0
I20240502 14:15:09.357890 32216 tablet_bootstrap.cc:654] T ebbe2bc571aa44d0a093819144ce49b1 P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:09.370597 32229 tablet.cc:1620] T ba860bfc9ad941ac994e336df805d404 P e4cfb8fc5a474177af6848a6efa13a27: MemRowSet was empty: no flush needed.
I20240502 14:15:09.394376 26283 catalog_manager.cc:5576] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097) does not have the latest schema for tablet ba860bfc9ad941ac994e336df805d404 (table rebalancer_dynamic_table [id=c7b6f3bedc784c2eb7a081c8626ec47d]). Expected version 1 got 0
I20240502 14:15:09.390460 32230 tablet.cc:1722] T ba860bfc9ad941ac994e336df805d404 P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:09.404348 26283 catalog_manager.cc:5576] TS e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843) does not have the latest schema for tablet ba860bfc9ad941ac994e336df805d404 (table rebalancer_dynamic_table [id=c7b6f3bedc784c2eb7a081c8626ec47d]). Expected version 1 got 0
I20240502 14:15:09.404946 32230 tablet.cc:1620] T ba860bfc9ad941ac994e336df805d404 P 4092902b3e674857a0147f1b4f2cdb86: MemRowSet was empty: no flush needed.
I20240502 14:15:09.396178 32216 tablet_bootstrap.cc:492] T ebbe2bc571aa44d0a093819144ce49b1 P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:15:09.405303 32216 ts_tablet_manager.cc:1367] T ebbe2bc571aa44d0a093819144ce49b1 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.106s user 0.007s sys 0.000s
I20240502 14:15:09.406474 32216 raft_consensus.cc:348] T ebbe2bc571aa44d0a093819144ce49b1 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:09.413268 32216 raft_consensus.cc:374] T ebbe2bc571aa44d0a093819144ce49b1 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:09.413468 32216 raft_consensus.cc:729] T ebbe2bc571aa44d0a093819144ce49b1 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:15:09.413848 32216 consensus_queue.cc:260] T ebbe2bc571aa44d0a093819144ce49b1 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:09.418579 32216 ts_tablet_manager.cc:1398] T ebbe2bc571aa44d0a093819144ce49b1 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240502 14:15:09.423621 31749 raft_consensus.cc:482] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:09.423816 31749 raft_consensus.cc:504] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:09.424731 31749 leader_election.cc:288] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:15:09.426000 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebbe2bc571aa44d0a093819144ce49b1" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:15:09.426026 29176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebbe2bc571aa44d0a093819144ce49b1" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:15:09.426265 29176 raft_consensus.cc:2454] T ebbe2bc571aa44d0a093819144ce49b1 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 0.
I20240502 14:15:09.426283 29309 raft_consensus.cc:2454] T ebbe2bc571aa44d0a093819144ce49b1 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 0.
I20240502 14:15:09.426833 26493 leader_election.cc:302] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:15:09.427115 31749 raft_consensus.cc:2781] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:09.427222 31749 raft_consensus.cc:482] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:09.427289 31749 raft_consensus.cc:3037] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:09.431514 31749 raft_consensus.cc:504] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:09.435490 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebbe2bc571aa44d0a093819144ce49b1" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
I20240502 14:15:09.435878 29309 raft_consensus.cc:3037] T ebbe2bc571aa44d0a093819144ce49b1 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:09.432189 31749 leader_election.cc:288] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Requested vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:15:09.439519 29176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebbe2bc571aa44d0a093819144ce49b1" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:15:09.439787 29176 raft_consensus.cc:3037] T ebbe2bc571aa44d0a093819144ce49b1 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:09.442065 29309 raft_consensus.cc:2454] T ebbe2bc571aa44d0a093819144ce49b1 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:15:09.444859 26493 leader_election.cc:302] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:15:09.445508 31369 raft_consensus.cc:2781] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:09.445647 31369 raft_consensus.cc:686] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Becoming Leader. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Running, Role: LEADER
I20240502 14:15:09.445892 31369 consensus_queue.cc:237] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:09.451970 29176 raft_consensus.cc:2454] T ebbe2bc571aa44d0a093819144ce49b1 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:15:09.453131 31971 raft_consensus.cc:482] T 88b726d7142846d6893f4f3b763c0565 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:09.453421 31971 raft_consensus.cc:504] T 88b726d7142846d6893f4f3b763c0565 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:09.454313 31971 leader_election.cc:288] T 88b726d7142846d6893f4f3b763c0565 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:15:09.455523 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88b726d7142846d6893f4f3b763c0565" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:15:09.455843 29443 raft_consensus.cc:2454] T 88b726d7142846d6893f4f3b763c0565 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:15:09.456513 29511 leader_election.cc:302] T 88b726d7142846d6893f4f3b763c0565 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, 68a3c2b17ed94e06aba66ed4178132ab; no voters:
I20240502 14:15:09.456864 32137 raft_consensus.cc:2781] T 88b726d7142846d6893f4f3b763c0565 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:09.456965 32137 raft_consensus.cc:482] T 88b726d7142846d6893f4f3b763c0565 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:09.457180 32137 raft_consensus.cc:3037] T 88b726d7142846d6893f4f3b763c0565 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:09.457439 26819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88b726d7142846d6893f4f3b763c0565" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:15:09.457726 26819 raft_consensus.cc:2454] T 88b726d7142846d6893f4f3b763c0565 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:15:09.460786 32137 raft_consensus.cc:504] T 88b726d7142846d6893f4f3b763c0565 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:09.461059 26276 catalog_manager.cc:5484] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a reported cstate change: term changed from 0 to 1, leader changed from <none> to ba187e38a2754d60a12de9ac4856430a (127.19.94.194). New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:09.461793 26283 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:15:09.462118 26819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88b726d7142846d6893f4f3b763c0565" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:15:09.462340 26819 raft_consensus.cc:3037] T 88b726d7142846d6893f4f3b763c0565 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:09.462730 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88b726d7142846d6893f4f3b763c0565" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
I20240502 14:15:09.462945 29443 raft_consensus.cc:3037] T 88b726d7142846d6893f4f3b763c0565 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:09.464900 26819 raft_consensus.cc:2454] T 88b726d7142846d6893f4f3b763c0565 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:15:09.465451 29511 leader_election.cc:302] T 88b726d7142846d6893f4f3b763c0565 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:15:09.465668 29443 raft_consensus.cc:2454] T 88b726d7142846d6893f4f3b763c0565 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:15:09.465853 31971 raft_consensus.cc:2781] T 88b726d7142846d6893f4f3b763c0565 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:09.466044 31971 raft_consensus.cc:686] T 88b726d7142846d6893f4f3b763c0565 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Becoming Leader. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Running, Role: LEADER
I20240502 14:15:09.466344 31971 consensus_queue.cc:237] T 88b726d7142846d6893f4f3b763c0565 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:09.461475 32137 leader_election.cc:288] T 88b726d7142846d6893f4f3b763c0565 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Requested vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:15:09.473551 32227 tablet.cc:1722] T ba860bfc9ad941ac994e336df805d404 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:09.474076 32227 tablet.cc:1620] T ba860bfc9ad941ac994e336df805d404 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: MemRowSet was empty: no flush needed.
I20240502 14:15:09.481144 32230 tablet.cc:1722] T ba860bfc9ad941ac994e336df805d404 P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:09.481717 32230 tablet.cc:1620] T ba860bfc9ad941ac994e336df805d404 P 4092902b3e674857a0147f1b4f2cdb86: MemRowSet was empty: no flush needed.
I20240502 14:15:09.482991 32229 tablet.cc:1722] T ba860bfc9ad941ac994e336df805d404 P e4cfb8fc5a474177af6848a6efa13a27: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:09.483520 32229 tablet.cc:1620] T ba860bfc9ad941ac994e336df805d404 P e4cfb8fc5a474177af6848a6efa13a27: MemRowSet was empty: no flush needed.
I20240502 14:15:09.493852 26285 catalog_manager.cc:5484] T 88b726d7142846d6893f4f3b763c0565 P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200). New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:15:09.498145 26283 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=c7b6f3bedc784c2eb7a081c8626ec47d] alter complete (version 2)
I20240502 14:15:09.563177 31971 consensus_queue.cc:1035] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:09.589066 29309 raft_consensus.cc:3037] T 23f576e1bede478496122a9d1a2e7515 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:09.593889 31623 consensus_queue.cc:1035] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:09.608435 26283 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:15:09.608641 29442 consensus_queue.cc:237] T 177a34c96aa84cb4aa50bbd79cab4144 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:09.612767 26557 raft_consensus.cc:1261] T 177a34c96aa84cb4aa50bbd79cab4144 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:09.614328 32118 consensus_queue.cc:1035] T 177a34c96aa84cb4aa50bbd79cab4144 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:09.615859 29176 raft_consensus.cc:1261] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:09.620976 32181 consensus_queue.cc:1035] T 177a34c96aa84cb4aa50bbd79cab4144 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:09.625874 32227 tablet.cc:1722] T ba860bfc9ad941ac994e336df805d404 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:09.627959 32230 tablet.cc:1722] T ba860bfc9ad941ac994e336df805d404 P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:09.639024 32229 tablet.cc:1722] T ba860bfc9ad941ac994e336df805d404 P e4cfb8fc5a474177af6848a6efa13a27: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:09.656949 26285 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table_new_name [id=c7b6f3bedc784c2eb7a081c8626ec47d] alter complete (version 3)
I20240502 14:15:09.658295 32118 raft_consensus.cc:2932] T 177a34c96aa84cb4aa50bbd79cab4144 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:09.662101 29174 raft_consensus.cc:2932] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:09.667148 26557 raft_consensus.cc:2932] T 177a34c96aa84cb4aa50bbd79cab4144 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:09.681849 26283 catalog_manager.cc:5484] T 177a34c96aa84cb4aa50bbd79cab4144 P ba187e38a2754d60a12de9ac4856430a reported cstate change: config changed from index 2 to 3, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) evicted. New cstate: current_term: 1 leader_uuid: "68a3c2b17ed94e06aba66ed4178132ab" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:09.686838 26407 tablet_service.cc:1514] Processing DeleteTablet for tablet 177a34c96aa84cb4aa50bbd79cab4144 with delete_type TABLET_DATA_TOMBSTONED (TS 89af7f423cf44717a08d6b142f0ed49d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:37422
I20240502 14:15:09.701694 32265 tablet_replica.cc:330] T 177a34c96aa84cb4aa50bbd79cab4144 P 89af7f423cf44717a08d6b142f0ed49d: stopping tablet replica
I20240502 14:15:09.702065 32265 raft_consensus.cc:2229] T 177a34c96aa84cb4aa50bbd79cab4144 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:09.702368 32265 raft_consensus.cc:2258] T 177a34c96aa84cb4aa50bbd79cab4144 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:09.706494 32265 ts_tablet_manager.cc:1875] T 177a34c96aa84cb4aa50bbd79cab4144 P 89af7f423cf44717a08d6b142f0ed49d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:09.716014 32265 ts_tablet_manager.cc:1888] T 177a34c96aa84cb4aa50bbd79cab4144 P 89af7f423cf44717a08d6b142f0ed49d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:15:09.716279 32265 log.cc:1196] T 177a34c96aa84cb4aa50bbd79cab4144 P 89af7f423cf44717a08d6b142f0ed49d: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/177a34c96aa84cb4aa50bbd79cab4144
I20240502 14:15:09.717994 26271 catalog_manager.cc:4830] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): tablet 177a34c96aa84cb4aa50bbd79cab4144 (table rebalancer_extra_table_0 [id=7727fea7ae174c949e336e765c50e53d]) successfully deleted
I20240502 14:15:09.758332 26283 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:09.758577 26283 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:09.763422 26283 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet ba860bfc9ad941ac994e336df805d404
I20240502 14:15:09.764451 26669 tablet_service.cc:1514] Processing DeleteTablet for tablet ba860bfc9ad941ac994e336df805d404 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:09 UTC) from {username='slave'} at 127.0.0.1:34150
I20240502 14:15:09.765062 26797 tablet_service.cc:1514] Processing DeleteTablet for tablet ba860bfc9ad941ac994e336df805d404 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:09 UTC) from {username='slave'} at 127.0.0.1:56010
I20240502 14:15:09.765724 32278 tablet_replica.cc:330] T ba860bfc9ad941ac994e336df805d404 P e4cfb8fc5a474177af6848a6efa13a27: stopping tablet replica
I20240502 14:15:09.765890 29554 tablet_service.cc:1514] Processing DeleteTablet for tablet ba860bfc9ad941ac994e336df805d404 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:09 UTC) from {username='slave'} at 127.0.0.1:36398
I20240502 14:15:09.766088 32278 raft_consensus.cc:2229] T ba860bfc9ad941ac994e336df805d404 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:09.766577 32278 raft_consensus.cc:2258] T ba860bfc9ad941ac994e336df805d404 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:09.768745 32278 ts_tablet_manager.cc:1875] T ba860bfc9ad941ac994e336df805d404 P e4cfb8fc5a474177af6848a6efa13a27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:09.775228 32278 ts_tablet_manager.cc:1888] T ba860bfc9ad941ac994e336df805d404 P e4cfb8fc5a474177af6848a6efa13a27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:09.777721 32278 log.cc:1196] T ba860bfc9ad941ac994e336df805d404 P e4cfb8fc5a474177af6848a6efa13a27: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/wals/ba860bfc9ad941ac994e336df805d404
I20240502 14:15:09.778298 32278 ts_tablet_manager.cc:1909] T ba860bfc9ad941ac994e336df805d404 P e4cfb8fc5a474177af6848a6efa13a27: Deleting consensus metadata
I20240502 14:15:09.782665 26272 catalog_manager.cc:4830] TS e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): tablet ba860bfc9ad941ac994e336df805d404 (table rebalancer_dynamic_table_new_name [id=c7b6f3bedc784c2eb7a081c8626ec47d]) successfully deleted
I20240502 14:15:09.792838 32279 tablet_replica.cc:330] T ba860bfc9ad941ac994e336df805d404 P 4092902b3e674857a0147f1b4f2cdb86: stopping tablet replica
I20240502 14:15:09.792966 32280 tablet_replica.cc:330] T ba860bfc9ad941ac994e336df805d404 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: stopping tablet replica
I20240502 14:15:09.793182 32280 raft_consensus.cc:2229] T ba860bfc9ad941ac994e336df805d404 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:15:09.793200 32279 raft_consensus.cc:2229] T ba860bfc9ad941ac994e336df805d404 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:09.793606 32280 raft_consensus.cc:2258] T ba860bfc9ad941ac994e336df805d404 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:09.793453 32279 raft_consensus.cc:2258] T ba860bfc9ad941ac994e336df805d404 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:09.794673 32280 ts_tablet_manager.cc:1875] T ba860bfc9ad941ac994e336df805d404 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:09.795058 32279 ts_tablet_manager.cc:1875] T ba860bfc9ad941ac994e336df805d404 P 4092902b3e674857a0147f1b4f2cdb86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:09.801391 32280 ts_tablet_manager.cc:1888] T ba860bfc9ad941ac994e336df805d404 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:09.801820 32280 log.cc:1196] T ba860bfc9ad941ac994e336df805d404 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/wals/ba860bfc9ad941ac994e336df805d404
I20240502 14:15:09.801836 32279 ts_tablet_manager.cc:1888] T ba860bfc9ad941ac994e336df805d404 P 4092902b3e674857a0147f1b4f2cdb86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:09.801963 32279 log.cc:1196] T ba860bfc9ad941ac994e336df805d404 P 4092902b3e674857a0147f1b4f2cdb86: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/wal/wals/ba860bfc9ad941ac994e336df805d404
I20240502 14:15:09.802299 32280 ts_tablet_manager.cc:1909] T ba860bfc9ad941ac994e336df805d404 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting consensus metadata
I20240502 14:15:09.802578 32279 ts_tablet_manager.cc:1909] T ba860bfc9ad941ac994e336df805d404 P 4092902b3e674857a0147f1b4f2cdb86: Deleting consensus metadata
I20240502 14:15:09.803582 26272 catalog_manager.cc:4830] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): tablet ba860bfc9ad941ac994e336df805d404 (table rebalancer_dynamic_table_new_name [id=c7b6f3bedc784c2eb7a081c8626ec47d]) successfully deleted
I20240502 14:15:09.803855 26272 catalog_manager.cc:4830] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097): tablet ba860bfc9ad941ac994e336df805d404 (table rebalancer_dynamic_table_new_name [id=c7b6f3bedc784c2eb7a081c8626ec47d]) successfully deleted
I20240502 14:15:09.859303 32234 consensus_queue.cc:1035] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:09.873795 26277 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:15:09.888335 29554 tablet_service.cc:1467] Processing CreateTablet for tablet 4dc12ed13df8404494f5b1ac7bd51688 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9291f82c316b4a269154c8b465916181]), partition=
I20240502 14:15:09.888860 29554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dc12ed13df8404494f5b1ac7bd51688. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:09.899410 29287 tablet_service.cc:1467] Processing CreateTablet for tablet 4dc12ed13df8404494f5b1ac7bd51688 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9291f82c316b4a269154c8b465916181]), partition=
I20240502 14:15:09.899391 26407 tablet_service.cc:1467] Processing CreateTablet for tablet 4dc12ed13df8404494f5b1ac7bd51688 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9291f82c316b4a269154c8b465916181]), partition=
I20240502 14:15:09.899806 29287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dc12ed13df8404494f5b1ac7bd51688. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:09.899893 26407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dc12ed13df8404494f5b1ac7bd51688. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:09.904356 32234 consensus_queue.cc:1035] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:09.930353 32310 tablet_bootstrap.cc:492] T 4dc12ed13df8404494f5b1ac7bd51688 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:15:09.933264 32310 tablet_bootstrap.cc:654] T 4dc12ed13df8404494f5b1ac7bd51688 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:09.945012 32313 tablet_bootstrap.cc:492] T 4dc12ed13df8404494f5b1ac7bd51688 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:15:09.945173 32310 tablet_bootstrap.cc:492] T 4dc12ed13df8404494f5b1ac7bd51688 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:15:09.945456 32312 tablet_bootstrap.cc:492] T 4dc12ed13df8404494f5b1ac7bd51688 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:15:09.945590 32310 ts_tablet_manager.cc:1367] T 4dc12ed13df8404494f5b1ac7bd51688 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.007s
I20240502 14:15:09.946789 32310 raft_consensus.cc:348] T 4dc12ed13df8404494f5b1ac7bd51688 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:09.947285 32310 raft_consensus.cc:374] T 4dc12ed13df8404494f5b1ac7bd51688 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:09.947446 32310 raft_consensus.cc:729] T 4dc12ed13df8404494f5b1ac7bd51688 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:15:09.947825 32310 consensus_queue.cc:260] T 4dc12ed13df8404494f5b1ac7bd51688 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:09.949030 32310 ts_tablet_manager.cc:1398] T 4dc12ed13df8404494f5b1ac7bd51688 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:15:09.955315 32312 tablet_bootstrap.cc:654] T 4dc12ed13df8404494f5b1ac7bd51688 P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:09.955585 32313 tablet_bootstrap.cc:654] T 4dc12ed13df8404494f5b1ac7bd51688 P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:09.972016 32312 tablet_bootstrap.cc:492] T 4dc12ed13df8404494f5b1ac7bd51688 P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:15:09.972267 32312 ts_tablet_manager.cc:1367] T 4dc12ed13df8404494f5b1ac7bd51688 P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.000s
I20240502 14:15:09.973359 32312 raft_consensus.cc:348] T 4dc12ed13df8404494f5b1ac7bd51688 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:09.973650 32312 raft_consensus.cc:374] T 4dc12ed13df8404494f5b1ac7bd51688 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:09.973779 32312 raft_consensus.cc:729] T 4dc12ed13df8404494f5b1ac7bd51688 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:15:09.974035 32312 consensus_queue.cc:260] T 4dc12ed13df8404494f5b1ac7bd51688 P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:09.974496 31623 consensus_queue.cc:1035] T 88b726d7142846d6893f4f3b763c0565 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:09.976001 32312 ts_tablet_manager.cc:1398] T 4dc12ed13df8404494f5b1ac7bd51688 P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:15:09.978404 32313 tablet_bootstrap.cc:492] T 4dc12ed13df8404494f5b1ac7bd51688 P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:15:09.979146 32313 ts_tablet_manager.cc:1367] T 4dc12ed13df8404494f5b1ac7bd51688 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.005s
I20240502 14:15:09.980063 32313 raft_consensus.cc:348] T 4dc12ed13df8404494f5b1ac7bd51688 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:09.980370 32313 raft_consensus.cc:374] T 4dc12ed13df8404494f5b1ac7bd51688 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:09.980552 32313 raft_consensus.cc:729] T 4dc12ed13df8404494f5b1ac7bd51688 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:15:09.980835 32313 consensus_queue.cc:260] T 4dc12ed13df8404494f5b1ac7bd51688 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:09.981664 32313 ts_tablet_manager.cc:1398] T 4dc12ed13df8404494f5b1ac7bd51688 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:15:09.992372 32137 consensus_queue.cc:1035] T 88b726d7142846d6893f4f3b763c0565 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:10.033421 26284 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_extra_table_40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:10.050586 26407 tablet_service.cc:1467] Processing CreateTablet for tablet c9c7cac98ef14eab82e7a6f4d4d21f51 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=ede5d7f4445942ccba7beb577ec03516]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:10.050773 29287 tablet_service.cc:1467] Processing CreateTablet for tablet c9c7cac98ef14eab82e7a6f4d4d21f51 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=ede5d7f4445942ccba7beb577ec03516]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:10.051244 26407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9c7cac98ef14eab82e7a6f4d4d21f51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:10.051249 29287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9c7cac98ef14eab82e7a6f4d4d21f51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:10.059773 26669 tablet_service.cc:1467] Processing CreateTablet for tablet 870100caa8bf4ca1b49337ba374b1f83 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=ede5d7f4445942ccba7beb577ec03516]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:10.060273 26669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 870100caa8bf4ca1b49337ba374b1f83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:10.060359 26406 tablet_service.cc:1467] Processing CreateTablet for tablet 870100caa8bf4ca1b49337ba374b1f83 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=ede5d7f4445942ccba7beb577ec03516]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:10.060818 26406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 870100caa8bf4ca1b49337ba374b1f83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:10.061724 32310 tablet_bootstrap.cc:492] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:15:10.063923 32310 tablet_bootstrap.cc:654] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:10.066073 29421 tablet_service.cc:1467] Processing CreateTablet for tablet 870100caa8bf4ca1b49337ba374b1f83 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=ede5d7f4445942ccba7beb577ec03516]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:10.066648 29421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 870100caa8bf4ca1b49337ba374b1f83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:10.056226 26797 tablet_service.cc:1467] Processing CreateTablet for tablet c9c7cac98ef14eab82e7a6f4d4d21f51 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=ede5d7f4445942ccba7beb577ec03516]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:10.070279 26797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9c7cac98ef14eab82e7a6f4d4d21f51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:10.071106 29554 tablet_service.cc:1467] Processing CreateTablet for tablet f3f01a632ddf4810bd6572ff3e722a4b (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=ede5d7f4445942ccba7beb577ec03516]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:10.071538 29554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3f01a632ddf4810bd6572ff3e722a4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:10.079111 29155 tablet_service.cc:1467] Processing CreateTablet for tablet f3f01a632ddf4810bd6572ff3e722a4b (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=ede5d7f4445942ccba7beb577ec03516]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:10.079666 29155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3f01a632ddf4810bd6572ff3e722a4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:10.087261 32133 raft_consensus.cc:482] T 4dc12ed13df8404494f5b1ac7bd51688 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:10.087469 32133 raft_consensus.cc:504] T 4dc12ed13df8404494f5b1ac7bd51688 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:10.088524 32133 leader_election.cc:288] T 4dc12ed13df8404494f5b1ac7bd51688 P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:15:10.090584 26424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dc12ed13df8404494f5b1ac7bd51688" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:15:10.090911 26424 raft_consensus.cc:2454] T 4dc12ed13df8404494f5b1ac7bd51688 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 0.
I20240502 14:15:10.091426 29574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dc12ed13df8404494f5b1ac7bd51688" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86" is_pre_election: true
I20240502 14:15:10.091761 29574 raft_consensus.cc:2454] T 4dc12ed13df8404494f5b1ac7bd51688 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 0.
I20240502 14:15:10.092453 29244 leader_election.cc:302] T 4dc12ed13df8404494f5b1ac7bd51688 P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:15:10.094412 32330 tablet_bootstrap.cc:492] T 870100caa8bf4ca1b49337ba374b1f83 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:15:10.095189 32133 raft_consensus.cc:2781] T 4dc12ed13df8404494f5b1ac7bd51688 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:10.095337 32133 raft_consensus.cc:482] T 4dc12ed13df8404494f5b1ac7bd51688 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:10.095448 32133 raft_consensus.cc:3037] T 4dc12ed13df8404494f5b1ac7bd51688 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:10.103439 32133 raft_consensus.cc:504] T 4dc12ed13df8404494f5b1ac7bd51688 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:10.107625 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dc12ed13df8404494f5b1ac7bd51688" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:15:10.108350 26426 raft_consensus.cc:3037] T 4dc12ed13df8404494f5b1ac7bd51688 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:10.108827 29574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dc12ed13df8404494f5b1ac7bd51688" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86"
I20240502 14:15:10.109216 29574 raft_consensus.cc:3037] T 4dc12ed13df8404494f5b1ac7bd51688 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:10.111490 26426 raft_consensus.cc:2454] T 4dc12ed13df8404494f5b1ac7bd51688 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 1.
I20240502 14:15:10.112803 29244 leader_election.cc:302] T 4dc12ed13df8404494f5b1ac7bd51688 P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:15:10.113593 32119 raft_consensus.cc:2781] T 4dc12ed13df8404494f5b1ac7bd51688 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:10.114653 29574 raft_consensus.cc:2454] T 4dc12ed13df8404494f5b1ac7bd51688 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 1.
I20240502 14:15:10.126333 32330 tablet_bootstrap.cc:654] T 870100caa8bf4ca1b49337ba374b1f83 P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:10.138558 32330 tablet_bootstrap.cc:492] T 870100caa8bf4ca1b49337ba374b1f83 P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:15:10.143399 32339 tablet_bootstrap.cc:492] T f3f01a632ddf4810bd6572ff3e722a4b P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:15:10.144642 32312 tablet_bootstrap.cc:492] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:15:10.146522 32312 tablet_bootstrap.cc:654] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:10.139420 29421 tablet_service.cc:1467] Processing CreateTablet for tablet f3f01a632ddf4810bd6572ff3e722a4b (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=ede5d7f4445942ccba7beb577ec03516]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:10.147655 29421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3f01a632ddf4810bd6572ff3e722a4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:10.149828 32313 tablet_bootstrap.cc:492] T c9c7cac98ef14eab82e7a6f4d4d21f51 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:15:10.151726 32313 tablet_bootstrap.cc:654] T c9c7cac98ef14eab82e7a6f4d4d21f51 P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:10.157114 32330 ts_tablet_manager.cc:1367] T 870100caa8bf4ca1b49337ba374b1f83 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.063s user 0.000s sys 0.006s
I20240502 14:15:10.157853 32312 tablet_bootstrap.cc:492] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:15:10.158152 32312 ts_tablet_manager.cc:1367] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.003s
I20240502 14:15:10.161796 32312 raft_consensus.cc:348] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:10.162277 32312 raft_consensus.cc:374] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:10.162608 32312 raft_consensus.cc:729] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:15:10.163228 32312 consensus_queue.cc:260] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
W20240502 14:15:10.164274 29112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.199:41781, user_credentials={real_user=slave}} blocked reactor thread for 49382.1us
I20240502 14:15:10.164498 32330 raft_consensus.cc:348] T 870100caa8bf4ca1b49337ba374b1f83 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:10.164772 32330 raft_consensus.cc:374] T 870100caa8bf4ca1b49337ba374b1f83 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:10.164885 32330 raft_consensus.cc:729] T 870100caa8bf4ca1b49337ba374b1f83 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:15:10.165159 32119 raft_consensus.cc:686] T 4dc12ed13df8404494f5b1ac7bd51688 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 LEADER]: Becoming Leader. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Running, Role: LEADER
I20240502 14:15:10.166540 32339 tablet_bootstrap.cc:654] T f3f01a632ddf4810bd6572ff3e722a4b P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:10.166326 32119 consensus_queue.cc:237] T 4dc12ed13df8404494f5b1ac7bd51688 P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:10.169354 32330 consensus_queue.cc:260] T 870100caa8bf4ca1b49337ba374b1f83 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:10.170485 32312 ts_tablet_manager.cc:1398] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20240502 14:15:10.171010 32133 leader_election.cc:288] T 4dc12ed13df8404494f5b1ac7bd51688 P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 election: Requested vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:15:10.174772 32330 ts_tablet_manager.cc:1398] T 870100caa8bf4ca1b49337ba374b1f83 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.017s user 0.000s sys 0.002s
I20240502 14:15:10.177309 32330 tablet_bootstrap.cc:492] T f3f01a632ddf4810bd6572ff3e722a4b P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:15:10.187296 32339 tablet_bootstrap.cc:492] T f3f01a632ddf4810bd6572ff3e722a4b P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:15:10.188791 32339 ts_tablet_manager.cc:1367] T f3f01a632ddf4810bd6572ff3e722a4b P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.046s user 0.005s sys 0.000s
I20240502 14:15:10.189967 32330 tablet_bootstrap.cc:654] T f3f01a632ddf4810bd6572ff3e722a4b P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:10.190968 32339 raft_consensus.cc:348] T f3f01a632ddf4810bd6572ff3e722a4b P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:10.194648 32339 raft_consensus.cc:374] T f3f01a632ddf4810bd6572ff3e722a4b P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:10.195029 32339 raft_consensus.cc:729] T f3f01a632ddf4810bd6572ff3e722a4b P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:15:10.196008 32339 consensus_queue.cc:260] T f3f01a632ddf4810bd6572ff3e722a4b P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:10.196246 26277 catalog_manager.cc:5484] T 4dc12ed13df8404494f5b1ac7bd51688 P c1100e16b3b9495ca61d4f95e89b6a89 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198). New cstate: current_term: 1 leader_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:15:10.200534 32313 tablet_bootstrap.cc:492] T c9c7cac98ef14eab82e7a6f4d4d21f51 P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:15:10.200740 32313 ts_tablet_manager.cc:1367] T c9c7cac98ef14eab82e7a6f4d4d21f51 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.051s user 0.001s sys 0.004s
I20240502 14:15:10.201515 32313 raft_consensus.cc:348] T c9c7cac98ef14eab82e7a6f4d4d21f51 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:10.201678 32310 tablet_bootstrap.cc:492] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:15:10.201769 32313 raft_consensus.cc:374] T c9c7cac98ef14eab82e7a6f4d4d21f51 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:10.201864 32310 ts_tablet_manager.cc:1367] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.140s user 0.004s sys 0.018s
I20240502 14:15:10.201865 32313 raft_consensus.cc:729] T c9c7cac98ef14eab82e7a6f4d4d21f51 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:15:10.202703 32310 raft_consensus.cc:348] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:10.203008 32310 raft_consensus.cc:374] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:10.203119 32310 raft_consensus.cc:729] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:15:10.203075 32313 consensus_queue.cc:260] T c9c7cac98ef14eab82e7a6f4d4d21f51 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:10.203456 32310 consensus_queue.cc:260] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:10.203644 32313 ts_tablet_manager.cc:1398] T c9c7cac98ef14eab82e7a6f4d4d21f51 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:15:10.206081 32310 ts_tablet_manager.cc:1398] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20240502 14:15:10.206467 32310 tablet_bootstrap.cc:492] T 870100caa8bf4ca1b49337ba374b1f83 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:15:10.208614 32310 tablet_bootstrap.cc:654] T 870100caa8bf4ca1b49337ba374b1f83 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:10.210027 32332 tablet_bootstrap.cc:492] T c9c7cac98ef14eab82e7a6f4d4d21f51 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:15:10.210572 32339 ts_tablet_manager.cc:1398] T f3f01a632ddf4810bd6572ff3e722a4b P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20240502 14:15:10.218971 32040 raft_consensus.cc:482] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:10.219220 32040 raft_consensus.cc:504] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:10.220324 32331 tablet_bootstrap.cc:492] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:15:10.220458 32040 leader_election.cc:288] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:15:10.220930 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9c7cac98ef14eab82e7a6f4d4d21f51" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:15:10.221197 29309 raft_consensus.cc:2454] T c9c7cac98ef14eab82e7a6f4d4d21f51 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:15:10.221295 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9c7cac98ef14eab82e7a6f4d4d21f51" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:15:10.221791 26362 leader_election.cc:302] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
W20240502 14:15:10.221925 26363 leader_election.cc:341] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:10.222676 32123 raft_consensus.cc:2781] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:10.222808 32123 raft_consensus.cc:482] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:10.222940 32123 raft_consensus.cc:3037] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:10.224035 32332 tablet_bootstrap.cc:654] T c9c7cac98ef14eab82e7a6f4d4d21f51 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:10.225546 32331 tablet_bootstrap.cc:654] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:10.227228 32123 raft_consensus.cc:504] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:10.228658 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9c7cac98ef14eab82e7a6f4d4d21f51" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:15:10.228682 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9c7cac98ef14eab82e7a6f4d4d21f51" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
I20240502 14:15:10.228928 29309 raft_consensus.cc:3037] T c9c7cac98ef14eab82e7a6f4d4d21f51 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:10.228943 32331 tablet_bootstrap.cc:492] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:15:10.229079 32331 ts_tablet_manager.cc:1367] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.003s
W20240502 14:15:10.229244 26363 leader_election.cc:341] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:10.229776 32331 raft_consensus.cc:348] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:10.229960 32331 raft_consensus.cc:374] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:10.230010 32331 raft_consensus.cc:729] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:15:10.230216 32331 consensus_queue.cc:260] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:10.230898 32123 leader_election.cc:288] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Requested vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:15:10.231756 29309 raft_consensus.cc:2454] T c9c7cac98ef14eab82e7a6f4d4d21f51 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:15:10.231983 32331 ts_tablet_manager.cc:1398] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:15:10.232839 26362 leader_election.cc:302] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, c1100e16b3b9495ca61d4f95e89b6a89; no voters: e4cfb8fc5a474177af6848a6efa13a27
I20240502 14:15:10.233417 32123 raft_consensus.cc:2781] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:10.234100 32123 raft_consensus.cc:686] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Becoming Leader. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Running, Role: LEADER
I20240502 14:15:10.246824 32123 consensus_queue.cc:237] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:10.249629 32332 tablet_bootstrap.cc:492] T c9c7cac98ef14eab82e7a6f4d4d21f51 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:15:10.249950 32332 ts_tablet_manager.cc:1367] T c9c7cac98ef14eab82e7a6f4d4d21f51 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.000s
I20240502 14:15:10.250838 32332 raft_consensus.cc:348] T c9c7cac98ef14eab82e7a6f4d4d21f51 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:10.251164 32332 raft_consensus.cc:374] T c9c7cac98ef14eab82e7a6f4d4d21f51 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:10.252241 32332 raft_consensus.cc:729] T c9c7cac98ef14eab82e7a6f4d4d21f51 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:15:10.252677 32332 consensus_queue.cc:260] T c9c7cac98ef14eab82e7a6f4d4d21f51 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:10.258252 32332 ts_tablet_manager.cc:1398] T c9c7cac98ef14eab82e7a6f4d4d21f51 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 14:15:10.273905 32330 tablet_bootstrap.cc:492] T f3f01a632ddf4810bd6572ff3e722a4b P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:15:10.274320 32330 ts_tablet_manager.cc:1367] T f3f01a632ddf4810bd6572ff3e722a4b P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.097s user 0.005s sys 0.000s
I20240502 14:15:10.275620 32330 raft_consensus.cc:348] T f3f01a632ddf4810bd6572ff3e722a4b P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:10.276007 32330 raft_consensus.cc:374] T f3f01a632ddf4810bd6572ff3e722a4b P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:10.276255 32330 raft_consensus.cc:729] T f3f01a632ddf4810bd6572ff3e722a4b P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:15:10.276786 32330 consensus_queue.cc:260] T f3f01a632ddf4810bd6572ff3e722a4b P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:10.279654 32330 ts_tablet_manager.cc:1398] T f3f01a632ddf4810bd6572ff3e722a4b P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 14:15:10.281841 32310 tablet_bootstrap.cc:492] T 870100caa8bf4ca1b49337ba374b1f83 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:15:10.284227 32310 ts_tablet_manager.cc:1367] T 870100caa8bf4ca1b49337ba374b1f83 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.078s user 0.000s sys 0.004s
I20240502 14:15:10.291800 32310 raft_consensus.cc:348] T 870100caa8bf4ca1b49337ba374b1f83 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:10.294325 26277 catalog_manager.cc:5484] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: term changed from 0 to 1, leader changed from <none> to 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193). New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:10.297472 32310 raft_consensus.cc:374] T 870100caa8bf4ca1b49337ba374b1f83 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:10.297672 32310 raft_consensus.cc:729] T 870100caa8bf4ca1b49337ba374b1f83 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:15:10.298069 32310 consensus_queue.cc:260] T 870100caa8bf4ca1b49337ba374b1f83 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:10.307714 32310 ts_tablet_manager.cc:1398] T 870100caa8bf4ca1b49337ba374b1f83 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.021s user 0.000s sys 0.002s
I20240502 14:15:10.315686 26277 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:15:10.342334 29574 raft_consensus.cc:1261] T 4dc12ed13df8404494f5b1ac7bd51688 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Refusing update from remote peer c1100e16b3b9495ca61d4f95e89b6a89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:10.342960 26426 raft_consensus.cc:1261] T 4dc12ed13df8404494f5b1ac7bd51688 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer c1100e16b3b9495ca61d4f95e89b6a89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:10.343022 32119 consensus_queue.cc:1035] T 4dc12ed13df8404494f5b1ac7bd51688 P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:10.344269 32119 consensus_queue.cc:1035] T 4dc12ed13df8404494f5b1ac7bd51688 P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:10.387514 26280 catalog_manager.cc:5576] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097) does not have the latest schema for tablet 4dc12ed13df8404494f5b1ac7bd51688 (table rebalancer_dynamic_table [id=9291f82c316b4a269154c8b465916181]). Expected version 1 got 0
I20240502 14:15:10.392650 32222 mvcc.cc:205] Tried to move back new op lower bound from 7023244535147184128 to 7023244534466772992. Current Snapshot: MvccSnapshot[applied={T|T < 7023244535147184128}]
I20240502 14:15:10.392930 32360 tablet.cc:1722] T 4dc12ed13df8404494f5b1ac7bd51688 P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:10.394308 32360 tablet.cc:1620] T 4dc12ed13df8404494f5b1ac7bd51688 P 4092902b3e674857a0147f1b4f2cdb86: MemRowSet was empty: no flush needed.
I20240502 14:15:10.397004 26280 catalog_manager.cc:5576] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097) does not have the latest schema for tablet 4dc12ed13df8404494f5b1ac7bd51688 (table rebalancer_dynamic_table [id=9291f82c316b4a269154c8b465916181]). Expected version 1 got 0
I20240502 14:15:10.401844 29155 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:10.398480 32358 tablet.cc:1722] T 4dc12ed13df8404494f5b1ac7bd51688 P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:10.406391 32358 tablet.cc:1620] T 4dc12ed13df8404494f5b1ac7bd51688 P c1100e16b3b9495ca61d4f95e89b6a89: MemRowSet was empty: no flush needed.
I20240502 14:15:10.426182 26277 catalog_manager.cc:5576] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985) does not have the latest schema for tablet 4dc12ed13df8404494f5b1ac7bd51688 (table rebalancer_dynamic_table [id=9291f82c316b4a269154c8b465916181]). Expected version 1 got 0
I20240502 14:15:10.430142 31617 raft_consensus.cc:482] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:10.430337 31617 raft_consensus.cc:504] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:10.431214 31617 leader_election.cc:288] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:15:10.432248 26424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "870100caa8bf4ca1b49337ba374b1f83" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:15:10.432538 26424 raft_consensus.cc:2454] T 870100caa8bf4ca1b49337ba374b1f83 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:15:10.433209 26624 leader_election.cc:302] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:15:10.433593 31617 raft_consensus.cc:2781] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:10.433681 31617 raft_consensus.cc:482] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:10.433740 31617 raft_consensus.cc:3037] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:10.436847 31617 raft_consensus.cc:504] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:10.438526 26424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "870100caa8bf4ca1b49337ba374b1f83" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:15:10.438776 26424 raft_consensus.cc:3037] T 870100caa8bf4ca1b49337ba374b1f83 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:10.441740 26424 raft_consensus.cc:2454] T 870100caa8bf4ca1b49337ba374b1f83 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:15:10.443842 26624 leader_election.cc:302] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:15:10.444149 32260 raft_consensus.cc:2781] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:10.453069 32137 raft_consensus.cc:482] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:10.453310 32137 raft_consensus.cc:504] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:10.454241 32137 leader_election.cc:288] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:15:10.455102 29176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3f01a632ddf4810bd6572ff3e722a4b" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:15:10.455401 29176 raft_consensus.cc:2454] T f3f01a632ddf4810bd6572ff3e722a4b P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:15:10.456133 29510 leader_election.cc:302] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 4092902b3e674857a0147f1b4f2cdb86; no voters:
I20240502 14:15:10.456521 32137 raft_consensus.cc:2781] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:10.456666 32137 raft_consensus.cc:482] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:10.456750 32137 raft_consensus.cc:3037] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:10.458417 32260 raft_consensus.cc:686] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Becoming Leader. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Running, Role: LEADER
I20240502 14:15:10.459216 26277 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=9291f82c316b4a269154c8b465916181] alter complete (version 1)
I20240502 14:15:10.460177 31617 leader_election.cc:288] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Requested vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:15:10.460217 32137 raft_consensus.cc:504] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:10.461515 29176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3f01a632ddf4810bd6572ff3e722a4b" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:15:10.461786 29176 raft_consensus.cc:3037] T f3f01a632ddf4810bd6572ff3e722a4b P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:10.465065 29176 raft_consensus.cc:2454] T f3f01a632ddf4810bd6572ff3e722a4b P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:15:10.467018 29510 leader_election.cc:302] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 4092902b3e674857a0147f1b4f2cdb86; no voters:
I20240502 14:15:10.467381 31623 raft_consensus.cc:2781] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:10.477708 32137 leader_election.cc:288] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Requested vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:15:10.479745 32368 tablet.cc:1722] T 4dc12ed13df8404494f5b1ac7bd51688 P 89af7f423cf44717a08d6b142f0ed49d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:10.488902 31623 raft_consensus.cc:686] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Becoming Leader. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Running, Role: LEADER
I20240502 14:15:10.489285 31623 consensus_queue.cc:237] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:10.492674 26277 catalog_manager.cc:5576] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985) does not have the latest schema for tablet 4dc12ed13df8404494f5b1ac7bd51688 (table rebalancer_dynamic_table [id=9291f82c316b4a269154c8b465916181]). Expected version 1 got 0
I20240502 14:15:10.498248 32368 tablet.cc:1620] T 4dc12ed13df8404494f5b1ac7bd51688 P 89af7f423cf44717a08d6b142f0ed49d: MemRowSet was empty: no flush needed.
I20240502 14:15:10.503013 26277 catalog_manager.cc:5484] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200). New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:15:10.485494 32260 consensus_queue.cc:237] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:10.524529 26277 catalog_manager.cc:5484] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195). New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:10.535218 29554 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:15:10.537722 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "870100caa8bf4ca1b49337ba374b1f83" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:15:10.538028 29443 raft_consensus.cc:2454] T 870100caa8bf4ca1b49337ba374b1f83 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:15:10.538736 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "870100caa8bf4ca1b49337ba374b1f83" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
I20240502 14:15:10.538959 29443 raft_consensus.cc:3037] T 870100caa8bf4ca1b49337ba374b1f83 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:10.542042 29443 raft_consensus.cc:2454] T 870100caa8bf4ca1b49337ba374b1f83 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:15:10.552837 29439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3f01a632ddf4810bd6572ff3e722a4b" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:15:10.553319 29439 raft_consensus.cc:2454] T f3f01a632ddf4810bd6572ff3e722a4b P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:15:10.552836 29442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3f01a632ddf4810bd6572ff3e722a4b" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
I20240502 14:15:10.553740 29442 raft_consensus.cc:3037] T f3f01a632ddf4810bd6572ff3e722a4b P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:10.558804 29442 raft_consensus.cc:2454] T f3f01a632ddf4810bd6572ff3e722a4b P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:15:10.567808 26669 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:15:10.569571 26277 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:15:10.584405 29421 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:15:10.596693 32358 tablet.cc:1722] T 4dc12ed13df8404494f5b1ac7bd51688 P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:10.598459 26406 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:15:10.599467 32360 tablet.cc:1722] T 4dc12ed13df8404494f5b1ac7bd51688 P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:10.600014 32360 tablet.cc:1620] T 4dc12ed13df8404494f5b1ac7bd51688 P 4092902b3e674857a0147f1b4f2cdb86: MemRowSet was empty: no flush needed.
I20240502 14:15:10.604909 32358 tablet.cc:1620] T 4dc12ed13df8404494f5b1ac7bd51688 P c1100e16b3b9495ca61d4f95e89b6a89: MemRowSet was empty: no flush needed.
I20240502 14:15:10.610400 32368 tablet.cc:1722] T 4dc12ed13df8404494f5b1ac7bd51688 P 89af7f423cf44717a08d6b142f0ed49d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:10.632939 26277 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=9291f82c316b4a269154c8b465916181] alter complete (version 2)
I20240502 14:15:10.635571 32368 tablet.cc:1620] T 4dc12ed13df8404494f5b1ac7bd51688 P 89af7f423cf44717a08d6b142f0ed49d: MemRowSet was empty: no flush needed.
I20240502 14:15:10.691627 29287 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:10.705338 26538 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:10.714322 26819 raft_consensus.cc:3037] T c9c7cac98ef14eab82e7a6f4d4d21f51 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:10.727425 32123 consensus_queue.cc:1035] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:10.735975 26277 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:15:10.736430 32040 consensus_queue.cc:1035] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:10.745357 32358 tablet.cc:1722] T 4dc12ed13df8404494f5b1ac7bd51688 P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:10.746482 32368 tablet.cc:1722] T 4dc12ed13df8404494f5b1ac7bd51688 P 89af7f423cf44717a08d6b142f0ed49d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:10.748381 32360 tablet.cc:1722] T 4dc12ed13df8404494f5b1ac7bd51688 P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:10.765152 26277 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table_new_name [id=9291f82c316b4a269154c8b465916181] alter complete (version 3)
I20240502 14:15:10.866931 26797 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:10.867755 26277 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:10.868037 26277 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:10.873759 26277 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet 4dc12ed13df8404494f5b1ac7bd51688
I20240502 14:15:10.875564 29287 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dc12ed13df8404494f5b1ac7bd51688 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:10 UTC) from {username='slave'} at 127.0.0.1:59160
I20240502 14:15:10.876533 26406 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dc12ed13df8404494f5b1ac7bd51688 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:10 UTC) from {username='slave'} at 127.0.0.1:37422
I20240502 14:15:10.883422 29554 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dc12ed13df8404494f5b1ac7bd51688 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:10 UTC) from {username='slave'} at 127.0.0.1:36398
I20240502 14:15:10.901767 32392 tablet_replica.cc:330] T 4dc12ed13df8404494f5b1ac7bd51688 P 4092902b3e674857a0147f1b4f2cdb86: stopping tablet replica
I20240502 14:15:10.902170 32391 tablet_replica.cc:330] T 4dc12ed13df8404494f5b1ac7bd51688 P 89af7f423cf44717a08d6b142f0ed49d: stopping tablet replica
I20240502 14:15:10.902477 32391 raft_consensus.cc:2229] T 4dc12ed13df8404494f5b1ac7bd51688 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:10.902805 32392 raft_consensus.cc:2229] T 4dc12ed13df8404494f5b1ac7bd51688 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:10.903098 32391 raft_consensus.cc:2258] T 4dc12ed13df8404494f5b1ac7bd51688 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:10.903374 32392 raft_consensus.cc:2258] T 4dc12ed13df8404494f5b1ac7bd51688 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:10.904124 32390 tablet_replica.cc:330] T 4dc12ed13df8404494f5b1ac7bd51688 P c1100e16b3b9495ca61d4f95e89b6a89: stopping tablet replica
I20240502 14:15:10.906867 32391 ts_tablet_manager.cc:1875] T 4dc12ed13df8404494f5b1ac7bd51688 P 89af7f423cf44717a08d6b142f0ed49d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:10.908738 32392 ts_tablet_manager.cc:1875] T 4dc12ed13df8404494f5b1ac7bd51688 P 4092902b3e674857a0147f1b4f2cdb86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:10.909143 32390 raft_consensus.cc:2229] T 4dc12ed13df8404494f5b1ac7bd51688 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:15:10.911335 32390 raft_consensus.cc:2258] T 4dc12ed13df8404494f5b1ac7bd51688 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:10.914520 32375 consensus_queue.cc:1035] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:10.939457 32391 ts_tablet_manager.cc:1888] T 4dc12ed13df8404494f5b1ac7bd51688 P 89af7f423cf44717a08d6b142f0ed49d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:10.941537 32391 log.cc:1196] T 4dc12ed13df8404494f5b1ac7bd51688 P 89af7f423cf44717a08d6b142f0ed49d: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/4dc12ed13df8404494f5b1ac7bd51688
I20240502 14:15:10.941504 32390 ts_tablet_manager.cc:1875] T 4dc12ed13df8404494f5b1ac7bd51688 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:10.945148 32391 ts_tablet_manager.cc:1909] T 4dc12ed13df8404494f5b1ac7bd51688 P 89af7f423cf44717a08d6b142f0ed49d: Deleting consensus metadata
I20240502 14:15:10.946255 26271 catalog_manager.cc:4830] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): tablet 4dc12ed13df8404494f5b1ac7bd51688 (table rebalancer_dynamic_table_new_name [id=9291f82c316b4a269154c8b465916181]) successfully deleted
I20240502 14:15:10.980712 31623 consensus_queue.cc:1035] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:10.988157 32392 ts_tablet_manager.cc:1888] T 4dc12ed13df8404494f5b1ac7bd51688 P 4092902b3e674857a0147f1b4f2cdb86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:10.988313 32392 log.cc:1196] T 4dc12ed13df8404494f5b1ac7bd51688 P 4092902b3e674857a0147f1b4f2cdb86: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/wal/wals/4dc12ed13df8404494f5b1ac7bd51688
I20240502 14:15:10.990154 32390 ts_tablet_manager.cc:1888] T 4dc12ed13df8404494f5b1ac7bd51688 P c1100e16b3b9495ca61d4f95e89b6a89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:10.990341 32390 log.cc:1196] T 4dc12ed13df8404494f5b1ac7bd51688 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal/wals/4dc12ed13df8404494f5b1ac7bd51688
I20240502 14:15:10.991175 32392 ts_tablet_manager.cc:1909] T 4dc12ed13df8404494f5b1ac7bd51688 P 4092902b3e674857a0147f1b4f2cdb86: Deleting consensus metadata
I20240502 14:15:10.991921 32390 ts_tablet_manager.cc:1909] T 4dc12ed13df8404494f5b1ac7bd51688 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting consensus metadata
I20240502 14:15:10.994186 26272 catalog_manager.cc:4830] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097): tablet 4dc12ed13df8404494f5b1ac7bd51688 (table rebalancer_dynamic_table_new_name [id=9291f82c316b4a269154c8b465916181]) successfully deleted
I20240502 14:15:10.996018 26271 catalog_manager.cc:4830] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): tablet 4dc12ed13df8404494f5b1ac7bd51688 (table rebalancer_dynamic_table_new_name [id=9291f82c316b4a269154c8b465916181]) successfully deleted
I20240502 14:15:11.002956 26280 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:15:11.037289 32370 consensus_queue.cc:1035] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:11.047147 29287 tablet_service.cc:1467] Processing CreateTablet for tablet f46d576031a94b0d8bd30ff3dd242e19 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=91b2a76f31334a8cbb1666ed9eb1e7f3]), partition=
I20240502 14:15:11.047714 29287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f46d576031a94b0d8bd30ff3dd242e19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:11.054550 29155 tablet_service.cc:1467] Processing CreateTablet for tablet f46d576031a94b0d8bd30ff3dd242e19 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=91b2a76f31334a8cbb1666ed9eb1e7f3]), partition=
I20240502 14:15:11.055140 29155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f46d576031a94b0d8bd30ff3dd242e19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:11.056850 26669 tablet_service.cc:1467] Processing CreateTablet for tablet f46d576031a94b0d8bd30ff3dd242e19 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=91b2a76f31334a8cbb1666ed9eb1e7f3]), partition=
I20240502 14:15:11.057330 26669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f46d576031a94b0d8bd30ff3dd242e19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:11.091210 26277 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_extra_table_42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:11.126967 32370 consensus_queue.cc:1035] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:11.130677 26668 tablet_service.cc:1467] Processing CreateTablet for tablet 19f81d8b28564c0dbb0b80bd5018ed5a (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=0a0581a07e774016963810a988c20e50]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:11.131175 26668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19f81d8b28564c0dbb0b80bd5018ed5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:11.132356 26797 tablet_service.cc:1467] Processing CreateTablet for tablet 19f81d8b28564c0dbb0b80bd5018ed5a (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=0a0581a07e774016963810a988c20e50]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:11.132807 26797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19f81d8b28564c0dbb0b80bd5018ed5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:11.136884 29154 tablet_service.cc:1467] Processing CreateTablet for tablet 19f81d8b28564c0dbb0b80bd5018ed5a (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=0a0581a07e774016963810a988c20e50]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:11.137388 29154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19f81d8b28564c0dbb0b80bd5018ed5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:11.148998 32402 tablet_bootstrap.cc:492] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:15:11.151372 32402 tablet_bootstrap.cc:654] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:11.156634 26538 tablet_service.cc:1467] Processing CreateTablet for tablet 2e59791e95cc452ea30d1ed8bb6178ae (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=0a0581a07e774016963810a988c20e50]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:11.157135 26538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e59791e95cc452ea30d1ed8bb6178ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:11.175350 29288 tablet_service.cc:1467] Processing CreateTablet for tablet 2e59791e95cc452ea30d1ed8bb6178ae (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=0a0581a07e774016963810a988c20e50]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:11.175910 29288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e59791e95cc452ea30d1ed8bb6178ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:11.176494 32406 tablet_bootstrap.cc:492] T f46d576031a94b0d8bd30ff3dd242e19 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:15:11.176759 32405 tablet_bootstrap.cc:492] T 2e59791e95cc452ea30d1ed8bb6178ae P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:15:11.183861 26669 tablet_service.cc:1467] Processing CreateTablet for tablet 2e59791e95cc452ea30d1ed8bb6178ae (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=0a0581a07e774016963810a988c20e50]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:11.182096 32405 tablet_bootstrap.cc:654] T 2e59791e95cc452ea30d1ed8bb6178ae P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:11.184329 26669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e59791e95cc452ea30d1ed8bb6178ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:11.184474 32406 tablet_bootstrap.cc:654] T f46d576031a94b0d8bd30ff3dd242e19 P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:11.198297 29554 tablet_service.cc:1467] Processing CreateTablet for tablet 06283c819eaf404b89d3c93c2b080a70 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=0a0581a07e774016963810a988c20e50]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:11.198860 29554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06283c819eaf404b89d3c93c2b080a70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:11.203949 32406 tablet_bootstrap.cc:492] T f46d576031a94b0d8bd30ff3dd242e19 P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:15:11.204205 32406 ts_tablet_manager.cc:1367] T f46d576031a94b0d8bd30ff3dd242e19 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.000s
I20240502 14:15:11.205206 32406 raft_consensus.cc:348] T f46d576031a94b0d8bd30ff3dd242e19 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:11.205601 32406 raft_consensus.cc:374] T f46d576031a94b0d8bd30ff3dd242e19 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:11.205772 32406 raft_consensus.cc:729] T f46d576031a94b0d8bd30ff3dd242e19 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:15:11.206097 32406 consensus_queue.cc:260] T f46d576031a94b0d8bd30ff3dd242e19 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:11.207019 32406 ts_tablet_manager.cc:1398] T f46d576031a94b0d8bd30ff3dd242e19 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:15:11.207075 26538 tablet_service.cc:1467] Processing CreateTablet for tablet 06283c819eaf404b89d3c93c2b080a70 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=0a0581a07e774016963810a988c20e50]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:11.207612 26538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06283c819eaf404b89d3c93c2b080a70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:11.208071 32406 tablet_bootstrap.cc:492] T 2e59791e95cc452ea30d1ed8bb6178ae P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:15:11.215104 32406 tablet_bootstrap.cc:654] T 2e59791e95cc452ea30d1ed8bb6178ae P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:11.220666 29287 tablet_service.cc:1467] Processing CreateTablet for tablet 06283c819eaf404b89d3c93c2b080a70 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=0a0581a07e774016963810a988c20e50]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:11.221489 29287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06283c819eaf404b89d3c93c2b080a70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:11.224625 32406 tablet_bootstrap.cc:492] T 2e59791e95cc452ea30d1ed8bb6178ae P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:15:11.225410 32406 ts_tablet_manager.cc:1367] T 2e59791e95cc452ea30d1ed8bb6178ae P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.002s
I20240502 14:15:11.226624 32406 raft_consensus.cc:348] T 2e59791e95cc452ea30d1ed8bb6178ae P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:11.236457 32409 tablet_bootstrap.cc:492] T 06283c819eaf404b89d3c93c2b080a70 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:15:11.239001 32409 tablet_bootstrap.cc:654] T 06283c819eaf404b89d3c93c2b080a70 P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:11.244894 32406 raft_consensus.cc:374] T 2e59791e95cc452ea30d1ed8bb6178ae P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:11.245126 32406 raft_consensus.cc:729] T 2e59791e95cc452ea30d1ed8bb6178ae P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:15:11.248859 32406 consensus_queue.cc:260] T 2e59791e95cc452ea30d1ed8bb6178ae P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:11.250018 32406 ts_tablet_manager.cc:1398] T 2e59791e95cc452ea30d1ed8bb6178ae P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.024s user 0.000s sys 0.002s
I20240502 14:15:11.257927 32405 tablet_bootstrap.cc:492] T 2e59791e95cc452ea30d1ed8bb6178ae P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:15:11.258136 32405 ts_tablet_manager.cc:1367] T 2e59791e95cc452ea30d1ed8bb6178ae P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.081s user 0.009s sys 0.010s
I20240502 14:15:11.261226 32405 raft_consensus.cc:348] T 2e59791e95cc452ea30d1ed8bb6178ae P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:11.261484 32405 raft_consensus.cc:374] T 2e59791e95cc452ea30d1ed8bb6178ae P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:11.261576 32405 raft_consensus.cc:729] T 2e59791e95cc452ea30d1ed8bb6178ae P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:15:11.261721 32401 tablet_bootstrap.cc:492] T 19f81d8b28564c0dbb0b80bd5018ed5a P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:15:11.261854 32405 consensus_queue.cc:260] T 2e59791e95cc452ea30d1ed8bb6178ae P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:11.262444 32405 ts_tablet_manager.cc:1398] T 2e59791e95cc452ea30d1ed8bb6178ae P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 14:15:11.262789 32405 tablet_bootstrap.cc:492] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:15:11.270897 32401 tablet_bootstrap.cc:654] T 19f81d8b28564c0dbb0b80bd5018ed5a P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:11.272871 32401 tablet_bootstrap.cc:492] T 19f81d8b28564c0dbb0b80bd5018ed5a P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:15:11.273111 32401 ts_tablet_manager.cc:1367] T 19f81d8b28564c0dbb0b80bd5018ed5a P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240502 14:15:11.273926 32401 raft_consensus.cc:348] T 19f81d8b28564c0dbb0b80bd5018ed5a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:11.274228 32401 raft_consensus.cc:374] T 19f81d8b28564c0dbb0b80bd5018ed5a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:11.274344 32401 raft_consensus.cc:729] T 19f81d8b28564c0dbb0b80bd5018ed5a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:15:11.274641 32401 consensus_queue.cc:260] T 19f81d8b28564c0dbb0b80bd5018ed5a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:11.274847 32405 tablet_bootstrap.cc:654] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:11.276793 32401 ts_tablet_manager.cc:1398] T 19f81d8b28564c0dbb0b80bd5018ed5a P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:15:11.277139 32401 tablet_bootstrap.cc:492] T f46d576031a94b0d8bd30ff3dd242e19 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:15:11.277721 32406 tablet_bootstrap.cc:492] T 06283c819eaf404b89d3c93c2b080a70 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:15:11.279855 32406 tablet_bootstrap.cc:654] T 06283c819eaf404b89d3c93c2b080a70 P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:11.278810 32401 tablet_bootstrap.cc:654] T f46d576031a94b0d8bd30ff3dd242e19 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:11.328586 32333 raft_consensus.cc:482] T 2e59791e95cc452ea30d1ed8bb6178ae P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:11.328811 32333 raft_consensus.cc:504] T 2e59791e95cc452ea30d1ed8bb6178ae P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:11.329779 32333 leader_election.cc:288] T 2e59791e95cc452ea30d1ed8bb6178ae P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:15:11.332880 26689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e59791e95cc452ea30d1ed8bb6178ae" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
W20240502 14:15:11.334444 29245 leader_election.cc:341] T 2e59791e95cc452ea30d1ed8bb6178ae P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:11.347121 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e59791e95cc452ea30d1ed8bb6178ae" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:15:11.347805 26558 raft_consensus.cc:2454] T 2e59791e95cc452ea30d1ed8bb6178ae P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 0.
I20240502 14:15:11.348565 29244 leader_election.cc:302] T 2e59791e95cc452ea30d1ed8bb6178ae P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a, c1100e16b3b9495ca61d4f95e89b6a89; no voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23
I20240502 14:15:11.349628 32333 raft_consensus.cc:2781] T 2e59791e95cc452ea30d1ed8bb6178ae P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:11.349802 32333 raft_consensus.cc:482] T 2e59791e95cc452ea30d1ed8bb6178ae P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:11.349908 32333 raft_consensus.cc:3037] T 2e59791e95cc452ea30d1ed8bb6178ae P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:11.361152 32370 raft_consensus.cc:482] T 19f81d8b28564c0dbb0b80bd5018ed5a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:11.361346 32370 raft_consensus.cc:504] T 19f81d8b28564c0dbb0b80bd5018ed5a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:11.362087 32370 leader_election.cc:288] T 19f81d8b28564c0dbb0b80bd5018ed5a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:15:11.363346 29176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f81d8b28564c0dbb0b80bd5018ed5a" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
W20240502 14:15:11.364176 26624 leader_election.cc:341] T 19f81d8b28564c0dbb0b80bd5018ed5a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:11.372767 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f81d8b28564c0dbb0b80bd5018ed5a" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
W20240502 14:15:11.374974 26625 leader_election.cc:341] T 19f81d8b28564c0dbb0b80bd5018ed5a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:11.375164 26625 leader_election.cc:302] T 19f81d8b28564c0dbb0b80bd5018ed5a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters: 1cb67158133e4648a6eceba32d4dbba3, e4cfb8fc5a474177af6848a6efa13a27
I20240502 14:15:11.375690 32370 raft_consensus.cc:2726] T 19f81d8b28564c0dbb0b80bd5018ed5a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:15:11.386274 32333 raft_consensus.cc:504] T 2e59791e95cc452ea30d1ed8bb6178ae P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:11.392283 26688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e59791e95cc452ea30d1ed8bb6178ae" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
W20240502 14:15:11.393253 29245 leader_election.cc:341] T 2e59791e95cc452ea30d1ed8bb6178ae P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:11.393972 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e59791e95cc452ea30d1ed8bb6178ae" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:15:11.394332 26558 raft_consensus.cc:3037] T 2e59791e95cc452ea30d1ed8bb6178ae P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:11.405148 32333 leader_election.cc:288] T 2e59791e95cc452ea30d1ed8bb6178ae P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 election: Requested vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:15:11.416383 32402 tablet_bootstrap.cc:492] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:15:11.419131 32402 ts_tablet_manager.cc:1367] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.270s user 0.011s sys 0.028s
I20240502 14:15:11.421017 32402 raft_consensus.cc:348] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:11.429538 32402 raft_consensus.cc:374] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:11.430090 32402 raft_consensus.cc:729] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:15:11.430858 32402 consensus_queue.cc:260] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:11.432585 32402 ts_tablet_manager.cc:1398] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240502 14:15:11.565758 32404 tablet_bootstrap.cc:492] T f46d576031a94b0d8bd30ff3dd242e19 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:15:11.567533 32409 tablet_bootstrap.cc:492] T 06283c819eaf404b89d3c93c2b080a70 P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:15:11.567798 32409 ts_tablet_manager.cc:1367] T 06283c819eaf404b89d3c93c2b080a70 P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.332s user 0.049s sys 0.093s
I20240502 14:15:11.568918 32409 raft_consensus.cc:348] T 06283c819eaf404b89d3c93c2b080a70 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:11.569314 32409 raft_consensus.cc:374] T 06283c819eaf404b89d3c93c2b080a70 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:11.569921 32409 raft_consensus.cc:729] T 06283c819eaf404b89d3c93c2b080a70 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:15:11.570344 32409 consensus_queue.cc:260] T 06283c819eaf404b89d3c93c2b080a70 P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:11.571218 32409 ts_tablet_manager.cc:1398] T 06283c819eaf404b89d3c93c2b080a70 P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:15:11.577165 32405 tablet_bootstrap.cc:492] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:15:11.577351 32405 ts_tablet_manager.cc:1367] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.315s user 0.015s sys 0.056s
I20240502 14:15:11.578825 32405 raft_consensus.cc:348] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:11.579064 32405 raft_consensus.cc:374] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:11.579139 32405 raft_consensus.cc:729] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:15:11.579202 32404 tablet_bootstrap.cc:654] T f46d576031a94b0d8bd30ff3dd242e19 P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:11.579411 32405 consensus_queue.cc:260] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:11.579979 32405 ts_tablet_manager.cc:1398] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
W20240502 14:15:11.645745 29364 kernel_stack_watchdog.cc:198] Thread 29443 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:637 for 384ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa965036980 (unknown) at ??:0
@ 0x90ef77 __sanitizer::internal_sched_yield() at ??:?
@ 0x87462d __interceptor_pthread_create at ??:?
@ 0x7fa961ca4d11 kudu::Thread::StartThread() at ??:0
@ 0xa4d187 kudu::Thread::Create() at ??:?
@ 0x7fa961cc57ab kudu::ThreadPool::CreateThread() at ??:0
@ 0x7fa961cc13ad kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7fa961cc64ff kudu::ThreadPool::Submit() at ??:0
@ 0x7fa96ce65e87 kudu::log::Log::AppendThread::Wake() at ??:0
@ 0x7fa96ce72fbc kudu::log::Log::AsyncAppend() at ??:0
@ 0x7fa96ce73995 kudu::log::Log::AsyncAppendReplicates() at ??:0
@ 0x7fa96d47147d kudu::consensus::LogCache::AppendOperations() at ??:0
@ 0x7fa96d412b1b kudu::consensus::PeerMessageQueue::AppendOperations() at ??:0
@ 0x7fa96d4d4c05 kudu::consensus::RaftConsensus::UpdateReplica() at ??:0
@ 0x7fa96d4d1678 kudu::consensus::RaftConsensus::Update() at ??:0
@ 0x7fa9711b010e kudu::tserver::ConsensusServiceImpl::UpdateConsensus() at ??:0
W20240502 14:15:11.654901 29443 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.523s user 0.008s sys 0.002s
W20240502 14:15:11.655048 29443 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.523s user 0.008s sys 0.002s
I20240502 14:15:11.659428 32336 raft_consensus.cc:482] T f46d576031a94b0d8bd30ff3dd242e19 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:11.659703 32336 raft_consensus.cc:504] T f46d576031a94b0d8bd30ff3dd242e19 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:11.663399 29176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46d576031a94b0d8bd30ff3dd242e19" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:15:11.671097 32336 leader_election.cc:288] T f46d576031a94b0d8bd30ff3dd242e19 P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
W20240502 14:15:11.671909 29244 leader_election.cc:341] T f46d576031a94b0d8bd30ff3dd242e19 P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:11.541509 26558 raft_consensus.cc:2454] T 2e59791e95cc452ea30d1ed8bb6178ae P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 1.
I20240502 14:15:11.675894 32404 tablet_bootstrap.cc:492] T f46d576031a94b0d8bd30ff3dd242e19 P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:15:11.676081 32404 ts_tablet_manager.cc:1367] T f46d576031a94b0d8bd30ff3dd242e19 P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.110s user 0.000s sys 0.016s
I20240502 14:15:11.676646 29244 leader_election.cc:302] T 2e59791e95cc452ea30d1ed8bb6178ae P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a, c1100e16b3b9495ca61d4f95e89b6a89; no voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23
I20240502 14:15:11.677229 32336 raft_consensus.cc:2781] T 2e59791e95cc452ea30d1ed8bb6178ae P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:11.677117 32404 raft_consensus.cc:348] T f46d576031a94b0d8bd30ff3dd242e19 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:11.677387 32404 raft_consensus.cc:374] T f46d576031a94b0d8bd30ff3dd242e19 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:11.677415 32336 raft_consensus.cc:686] T 2e59791e95cc452ea30d1ed8bb6178ae P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 LEADER]: Becoming Leader. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Running, Role: LEADER
I20240502 14:15:11.677486 32404 raft_consensus.cc:729] T f46d576031a94b0d8bd30ff3dd242e19 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:15:11.677778 32336 consensus_queue.cc:237] T 2e59791e95cc452ea30d1ed8bb6178ae P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:11.684247 32432 raft_consensus.cc:482] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:11.684562 32432 raft_consensus.cc:504] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:11.685770 32432 leader_election.cc:288] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:15:11.686743 29308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06283c819eaf404b89d3c93c2b080a70" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
W20240502 14:15:11.687487 26493 leader_election.cc:341] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:11.695029 26687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46d576031a94b0d8bd30ff3dd242e19" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:15:11.698403 32404 consensus_queue.cc:260] T f46d576031a94b0d8bd30ff3dd242e19 P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:11.705080 32404 ts_tablet_manager.cc:1398] T f46d576031a94b0d8bd30ff3dd242e19 P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.029s user 0.002s sys 0.001s
I20240502 14:15:11.709476 32404 tablet_bootstrap.cc:492] T 19f81d8b28564c0dbb0b80bd5018ed5a P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:15:11.714028 32404 tablet_bootstrap.cc:654] T 19f81d8b28564c0dbb0b80bd5018ed5a P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:11.716450 32157 raft_consensus.cc:482] T f46d576031a94b0d8bd30ff3dd242e19 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:11.716645 32157 raft_consensus.cc:504] T f46d576031a94b0d8bd30ff3dd242e19 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:11.717505 32157 leader_election.cc:288] T f46d576031a94b0d8bd30ff3dd242e19 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:15:11.721568 32404 tablet_bootstrap.cc:492] T 19f81d8b28564c0dbb0b80bd5018ed5a P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:15:11.721773 32404 ts_tablet_manager.cc:1367] T 19f81d8b28564c0dbb0b80bd5018ed5a P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.004s
I20240502 14:15:11.722615 32404 raft_consensus.cc:348] T 19f81d8b28564c0dbb0b80bd5018ed5a P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:11.722847 32404 raft_consensus.cc:374] T 19f81d8b28564c0dbb0b80bd5018ed5a P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:11.722925 32404 raft_consensus.cc:729] T 19f81d8b28564c0dbb0b80bd5018ed5a P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:15:11.723169 32404 consensus_queue.cc:260] T 19f81d8b28564c0dbb0b80bd5018ed5a P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:11.723343 26688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46d576031a94b0d8bd30ff3dd242e19" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
W20240502 14:15:11.724129 29112 leader_election.cc:341] T f46d576031a94b0d8bd30ff3dd242e19 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:11.729440 29575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06283c819eaf404b89d3c93c2b080a70" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86" is_pre_election: true
I20240502 14:15:11.729708 29575 raft_consensus.cc:2454] T 06283c819eaf404b89d3c93c2b080a70 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 0.
I20240502 14:15:11.730183 32404 ts_tablet_manager.cc:1398] T 19f81d8b28564c0dbb0b80bd5018ed5a P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240502 14:15:11.749209 26494 leader_election.cc:302] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, ba187e38a2754d60a12de9ac4856430a; no voters: c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:15:11.749586 31369 raft_consensus.cc:2781] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:11.749699 31369 raft_consensus.cc:482] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:11.749830 31369 raft_consensus.cc:3037] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:11.757261 31369 raft_consensus.cc:504] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:11.759902 31369 leader_election.cc:288] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Requested vote from peers 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:15:11.777724 32401 tablet_bootstrap.cc:492] T f46d576031a94b0d8bd30ff3dd242e19 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:15:11.777931 32401 ts_tablet_manager.cc:1367] T f46d576031a94b0d8bd30ff3dd242e19 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.501s user 0.043s sys 0.074s
I20240502 14:15:11.778853 32401 raft_consensus.cc:348] T f46d576031a94b0d8bd30ff3dd242e19 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:11.779132 32401 raft_consensus.cc:374] T f46d576031a94b0d8bd30ff3dd242e19 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:11.779196 32401 raft_consensus.cc:729] T f46d576031a94b0d8bd30ff3dd242e19 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:15:11.779443 32401 consensus_queue.cc:260] T f46d576031a94b0d8bd30ff3dd242e19 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:11.780054 32401 ts_tablet_manager.cc:1398] T f46d576031a94b0d8bd30ff3dd242e19 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240502 14:15:11.780361 32401 tablet_bootstrap.cc:492] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:15:11.782446 32401 tablet_bootstrap.cc:654] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:11.785758 32401 tablet_bootstrap.cc:492] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:15:11.785915 32401 ts_tablet_manager.cc:1367] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 14:15:11.786815 32401 raft_consensus.cc:348] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:11.787048 32401 raft_consensus.cc:374] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:11.787106 32401 raft_consensus.cc:729] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:15:11.787334 32401 consensus_queue.cc:260] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:11.787703 29575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06283c819eaf404b89d3c93c2b080a70" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86"
I20240502 14:15:11.787918 32401 ts_tablet_manager.cc:1398] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:15:11.787946 29575 raft_consensus.cc:3037] T 06283c819eaf404b89d3c93c2b080a70 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:11.792276 29575 raft_consensus.cc:2454] T 06283c819eaf404b89d3c93c2b080a70 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:15:11.808218 32232 raft_consensus.cc:482] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:11.808476 32232 raft_consensus.cc:504] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:11.809747 32232 leader_election.cc:288] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:15:11.810307 29176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f81d8b28564c0dbb0b80bd5018ed5a" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:15:11.810647 29176 raft_consensus.cc:2454] T 19f81d8b28564c0dbb0b80bd5018ed5a P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 0.
I20240502 14:15:11.811051 26754 leader_election.cc:302] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:15:11.811323 32232 raft_consensus.cc:2781] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:11.811380 32232 raft_consensus.cc:482] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:11.811429 32232 raft_consensus.cc:3037] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:11.812505 26688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f81d8b28564c0dbb0b80bd5018ed5a" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:15:11.812788 26688 raft_consensus.cc:2454] T 19f81d8b28564c0dbb0b80bd5018ed5a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 0.
I20240502 14:15:11.813952 32232 raft_consensus.cc:504] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:11.815238 29176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f81d8b28564c0dbb0b80bd5018ed5a" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:15:11.815466 29176 raft_consensus.cc:3037] T 19f81d8b28564c0dbb0b80bd5018ed5a P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:11.816210 29306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46d576031a94b0d8bd30ff3dd242e19" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:15:11.816558 29306 raft_consensus.cc:2454] T f46d576031a94b0d8bd30ff3dd242e19 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 0.
I20240502 14:15:11.818076 29176 raft_consensus.cc:2454] T 19f81d8b28564c0dbb0b80bd5018ed5a P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
I20240502 14:15:11.818652 26754 leader_election.cc:302] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:15:11.818954 32366 raft_consensus.cc:2781] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:11.819056 32366 raft_consensus.cc:686] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Becoming Leader. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Running, Role: LEADER
I20240502 14:15:11.819316 32366 consensus_queue.cc:237] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:11.820358 32232 leader_election.cc:288] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Requested vote from peers 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:15:11.821245 26688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f81d8b28564c0dbb0b80bd5018ed5a" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:15:11.821513 26688 raft_consensus.cc:3037] T 19f81d8b28564c0dbb0b80bd5018ed5a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
W20240502 14:15:11.822444 29245 leader_election.cc:341] T f46d576031a94b0d8bd30ff3dd242e19 P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:11.822650 29245 leader_election.cc:302] T f46d576031a94b0d8bd30ff3dd242e19 P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c1100e16b3b9495ca61d4f95e89b6a89; no voters: 1cb67158133e4648a6eceba32d4dbba3, 69ab78b2a1b74d2bbf2cdeafa0a4af23
I20240502 14:15:11.827888 26688 raft_consensus.cc:2454] T 19f81d8b28564c0dbb0b80bd5018ed5a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
W20240502 14:15:11.878513 29245 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.94.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 179994us
I20240502 14:15:11.882720 32333 raft_consensus.cc:2726] T f46d576031a94b0d8bd30ff3dd242e19 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 14:15:11.917636 32406 thread.cc:636] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.637s user 0.111s sys 0.164s
W20240502 14:15:11.917776 32406 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 0.637s user 0.111s sys 0.164s
I20240502 14:15:11.918862 32406 tablet_bootstrap.cc:492] T 06283c819eaf404b89d3c93c2b080a70 P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:15:11.919031 32406 ts_tablet_manager.cc:1367] T 06283c819eaf404b89d3c93c2b080a70 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.641s user 0.113s sys 0.165s
I20240502 14:15:11.919962 32406 raft_consensus.cc:348] T 06283c819eaf404b89d3c93c2b080a70 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:11.920223 32406 raft_consensus.cc:374] T 06283c819eaf404b89d3c93c2b080a70 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:11.920313 32406 raft_consensus.cc:729] T 06283c819eaf404b89d3c93c2b080a70 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:15:11.920553 32406 consensus_queue.cc:260] T 06283c819eaf404b89d3c93c2b080a70 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:11.921113 32406 ts_tablet_manager.cc:1398] T 06283c819eaf404b89d3c93c2b080a70 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:15:11.929966 29305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06283c819eaf404b89d3c93c2b080a70" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
I20240502 14:15:11.930193 29111 leader_election.cc:302] T f46d576031a94b0d8bd30ff3dd242e19 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, c1100e16b3b9495ca61d4f95e89b6a89; no voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23
I20240502 14:15:11.930281 29305 raft_consensus.cc:3037] T 06283c819eaf404b89d3c93c2b080a70 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:11.930981 32157 raft_consensus.cc:2781] T f46d576031a94b0d8bd30ff3dd242e19 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:11.931068 32157 raft_consensus.cc:482] T f46d576031a94b0d8bd30ff3dd242e19 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:11.931155 32157 raft_consensus.cc:3037] T f46d576031a94b0d8bd30ff3dd242e19 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:11.935251 29305 raft_consensus.cc:2454] T 06283c819eaf404b89d3c93c2b080a70 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:15:11.935717 32157 raft_consensus.cc:504] T f46d576031a94b0d8bd30ff3dd242e19 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:11.937645 32157 leader_election.cc:288] T f46d576031a94b0d8bd30ff3dd242e19 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 election: Requested vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:15:11.938548 26688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46d576031a94b0d8bd30ff3dd242e19" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:15:11.938818 26688 raft_consensus.cc:3037] T f46d576031a94b0d8bd30ff3dd242e19 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:11.941947 26688 raft_consensus.cc:2454] T f46d576031a94b0d8bd30ff3dd242e19 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 1.
W20240502 14:15:11.942705 29245 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.200:46097, user_credentials={real_user=slave}} blocked reactor thread for 60215.5us
I20240502 14:15:11.944854 26493 leader_election.cc:302] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:15:11.945210 32234 raft_consensus.cc:2781] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:11.945331 32234 raft_consensus.cc:686] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Becoming Leader. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Running, Role: LEADER
I20240502 14:15:11.945274 29304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46d576031a94b0d8bd30ff3dd242e19" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
I20240502 14:15:11.945549 29304 raft_consensus.cc:3037] T f46d576031a94b0d8bd30ff3dd242e19 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:11.945559 32234 consensus_queue.cc:237] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:11.947418 29112 leader_election.cc:302] T f46d576031a94b0d8bd30ff3dd242e19 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters:
I20240502 14:15:11.947976 32334 raft_consensus.cc:2781] T f46d576031a94b0d8bd30ff3dd242e19 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:11.948089 32334 raft_consensus.cc:686] T f46d576031a94b0d8bd30ff3dd242e19 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 LEADER]: Becoming Leader. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Running, Role: LEADER
I20240502 14:15:11.948343 32334 consensus_queue.cc:237] T f46d576031a94b0d8bd30ff3dd242e19 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:11.949467 29304 raft_consensus.cc:2454] T f46d576031a94b0d8bd30ff3dd242e19 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 1.
I20240502 14:15:11.954900 26276 catalog_manager.cc:5484] T f46d576031a94b0d8bd30ff3dd242e19 P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197). New cstate: current_term: 1 leader_uuid: "1cb67158133e4648a6eceba32d4dbba3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:11.988185 26280 catalog_manager.cc:5484] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196). New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
W20240502 14:15:12.007150 29510 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.198:43659, user_credentials={real_user=slave}} blocked reactor thread for 62588.3us
I20240502 14:15:12.020356 26281 catalog_manager.cc:5484] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a reported cstate change: term changed from 0 to 1, leader changed from <none> to ba187e38a2754d60a12de9ac4856430a (127.19.94.194). New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:15:12.021224 32370 raft_consensus.cc:482] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:12.021365 32370 raft_consensus.cc:504] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:12.022724 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e59791e95cc452ea30d1ed8bb6178ae" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:15:12.023027 26558 raft_consensus.cc:2379] T 2e59791e95cc452ea30d1ed8bb6178ae P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in current term 1: Already voted for candidate c1100e16b3b9495ca61d4f95e89b6a89 in this term.
I20240502 14:15:12.024694 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e59791e95cc452ea30d1ed8bb6178ae" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:15:12.025384 26624 leader_election.cc:302] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters: ba187e38a2754d60a12de9ac4856430a, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:15:12.025774 32423 raft_consensus.cc:3037] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:12.033090 32423 raft_consensus.cc:2726] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:15:12.035784 32370 leader_election.cc:288] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:15:12.078271 26284 catalog_manager.cc:5484] T 2e59791e95cc452ea30d1ed8bb6178ae P c1100e16b3b9495ca61d4f95e89b6a89 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198). New cstate: current_term: 1 leader_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:15:12.089898 29575 consensus_queue.cc:237] T 2532ebd7bdf14876b74a26afd674a7ac P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:12.111378 26424 raft_consensus.cc:1261] T 2532ebd7bdf14876b74a26afd674a7ac P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:12.112644 32446 consensus_queue.cc:1035] T 2532ebd7bdf14876b74a26afd674a7ac P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:12.117723 29176 raft_consensus.cc:1261] T 2532ebd7bdf14876b74a26afd674a7ac P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:12.118691 32375 consensus_queue.cc:1035] T 2532ebd7bdf14876b74a26afd674a7ac P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:12.119585 32375 raft_consensus.cc:2932] T 2532ebd7bdf14876b74a26afd674a7ac P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:12.128957 26424 raft_consensus.cc:2932] T 2532ebd7bdf14876b74a26afd674a7ac P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:12.138821 29176 raft_consensus.cc:2932] T 2532ebd7bdf14876b74a26afd674a7ac P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:12.150206 32439 consensus_queue.cc:1035] T 2e59791e95cc452ea30d1ed8bb6178ae P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:12.152184 26284 catalog_manager.cc:5484] T 2532ebd7bdf14876b74a26afd674a7ac P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:12.159169 29421 tablet_service.cc:1514] Processing DeleteTablet for tablet 2532ebd7bdf14876b74a26afd674a7ac with delete_type TABLET_DATA_TOMBSTONED (TS 68a3c2b17ed94e06aba66ed4178132ab not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:40652
I20240502 14:15:12.175256 26281 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:15:12.209717 26689 raft_consensus.cc:1261] T f46d576031a94b0d8bd30ff3dd242e19 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:12.209859 29309 raft_consensus.cc:1261] T f46d576031a94b0d8bd30ff3dd242e19 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:12.210450 32157 consensus_queue.cc:1035] T f46d576031a94b0d8bd30ff3dd242e19 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:12.210947 32157 consensus_queue.cc:1035] T f46d576031a94b0d8bd30ff3dd242e19 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:12.238210 32444 consensus_queue.cc:1035] T 2e59791e95cc452ea30d1ed8bb6178ae P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:12.253111 26281 catalog_manager.cc:5576] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659) does not have the latest schema for tablet f46d576031a94b0d8bd30ff3dd242e19 (table rebalancer_dynamic_table [id=91b2a76f31334a8cbb1666ed9eb1e7f3]). Expected version 1 got 0
I20240502 14:15:12.261620 26281 catalog_manager.cc:5576] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659) does not have the latest schema for tablet f46d576031a94b0d8bd30ff3dd242e19 (table rebalancer_dynamic_table [id=91b2a76f31334a8cbb1666ed9eb1e7f3]). Expected version 1 got 0
I20240502 14:15:12.258440 32482 tablet.cc:1722] T f46d576031a94b0d8bd30ff3dd242e19 P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:12.267985 32482 tablet.cc:1620] T f46d576031a94b0d8bd30ff3dd242e19 P c1100e16b3b9495ca61d4f95e89b6a89: MemRowSet was empty: no flush needed.
I20240502 14:15:12.275957 32469 tablet_replica.cc:330] T 2532ebd7bdf14876b74a26afd674a7ac P 68a3c2b17ed94e06aba66ed4178132ab: stopping tablet replica
I20240502 14:15:12.276227 32469 raft_consensus.cc:2229] T 2532ebd7bdf14876b74a26afd674a7ac P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:12.280453 32469 raft_consensus.cc:2258] T 2532ebd7bdf14876b74a26afd674a7ac P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:12.281221 32469 ts_tablet_manager.cc:1875] T 2532ebd7bdf14876b74a26afd674a7ac P 68a3c2b17ed94e06aba66ed4178132ab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:12.285634 26284 catalog_manager.cc:5576] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325) does not have the latest schema for tablet f46d576031a94b0d8bd30ff3dd242e19 (table rebalancer_dynamic_table [id=91b2a76f31334a8cbb1666ed9eb1e7f3]). Expected version 1 got 0
I20240502 14:15:12.298853 32469 ts_tablet_manager.cc:1888] T 2532ebd7bdf14876b74a26afd674a7ac P 68a3c2b17ed94e06aba66ed4178132ab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:15:12.298995 32469 log.cc:1196] T 2532ebd7bdf14876b74a26afd674a7ac P 68a3c2b17ed94e06aba66ed4178132ab: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal/wals/2532ebd7bdf14876b74a26afd674a7ac
I20240502 14:15:12.303136 26272 catalog_manager.cc:4830] TS 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): tablet 2532ebd7bdf14876b74a26afd674a7ac (table rebalancer_extra_table_6 [id=fe880af4989f416d82ae711576173990]) successfully deleted
I20240502 14:15:12.320256 26284 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=91b2a76f31334a8cbb1666ed9eb1e7f3] alter complete (version 1)
I20240502 14:15:12.322572 32366 consensus_queue.cc:1035] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:12.330938 32480 tablet.cc:1722] T f46d576031a94b0d8bd30ff3dd242e19 P 1cb67158133e4648a6eceba32d4dbba3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
W20240502 14:15:12.331777 29111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.198:43659, user_credentials={real_user=slave}} blocked reactor thread for 73045.2us
I20240502 14:15:12.337711 32480 tablet.cc:1620] T f46d576031a94b0d8bd30ff3dd242e19 P 1cb67158133e4648a6eceba32d4dbba3: MemRowSet was empty: no flush needed.
I20240502 14:15:12.354589 32493 tablet.cc:1722] T f46d576031a94b0d8bd30ff3dd242e19 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:12.356716 26284 catalog_manager.cc:5576] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325) does not have the latest schema for tablet f46d576031a94b0d8bd30ff3dd242e19 (table rebalancer_dynamic_table [id=91b2a76f31334a8cbb1666ed9eb1e7f3]). Expected version 1 got 0
I20240502 14:15:12.361510 32493 tablet.cc:1620] T f46d576031a94b0d8bd30ff3dd242e19 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: MemRowSet was empty: no flush needed.
I20240502 14:15:12.366631 32432 consensus_queue.cc:1035] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:12.375916 32234 consensus_queue.cc:1035] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:12.386107 32232 consensus_queue.cc:1035] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:12.434523 26281 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:15:12.540812 32480 tablet.cc:1722] T f46d576031a94b0d8bd30ff3dd242e19 P 1cb67158133e4648a6eceba32d4dbba3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:12.541534 32480 tablet.cc:1620] T f46d576031a94b0d8bd30ff3dd242e19 P 1cb67158133e4648a6eceba32d4dbba3: MemRowSet was empty: no flush needed.
I20240502 14:15:12.550988 32493 tablet.cc:1722] T f46d576031a94b0d8bd30ff3dd242e19 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:12.551565 32493 tablet.cc:1620] T f46d576031a94b0d8bd30ff3dd242e19 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: MemRowSet was empty: no flush needed.
I20240502 14:15:12.558081 32482 tablet.cc:1722] T f46d576031a94b0d8bd30ff3dd242e19 P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:12.558650 32482 tablet.cc:1620] T f46d576031a94b0d8bd30ff3dd242e19 P c1100e16b3b9495ca61d4f95e89b6a89: MemRowSet was empty: no flush needed.
I20240502 14:15:12.573333 26277 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=91b2a76f31334a8cbb1666ed9eb1e7f3] alter complete (version 2)
I20240502 14:15:12.613641 26281 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_extra_table_44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:12.627221 26406 tablet_service.cc:1467] Processing CreateTablet for tablet 112821e7380e41dcac66a38e21ba809d (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=d20a7e521c7f4f93a17f7a49aa1c5d28]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:12.627722 26406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 112821e7380e41dcac66a38e21ba809d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:12.627709 29421 tablet_service.cc:1467] Processing CreateTablet for tablet 112821e7380e41dcac66a38e21ba809d (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=d20a7e521c7f4f93a17f7a49aa1c5d28]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:12.628118 29421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 112821e7380e41dcac66a38e21ba809d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:12.634155 29554 tablet_service.cc:1467] Processing CreateTablet for tablet 112821e7380e41dcac66a38e21ba809d (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=d20a7e521c7f4f93a17f7a49aa1c5d28]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:12.634313 26407 tablet_service.cc:1467] Processing CreateTablet for tablet b0316714f34d44a5b537cdf20af42ce9 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=d20a7e521c7f4f93a17f7a49aa1c5d28]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:12.634649 29554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 112821e7380e41dcac66a38e21ba809d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:12.634725 26407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0316714f34d44a5b537cdf20af42ce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:12.636778 26797 tablet_service.cc:1467] Processing CreateTablet for tablet b0316714f34d44a5b537cdf20af42ce9 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=d20a7e521c7f4f93a17f7a49aa1c5d28]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:12.637197 26797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0316714f34d44a5b537cdf20af42ce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:12.640540 26538 tablet_service.cc:1467] Processing CreateTablet for tablet b0316714f34d44a5b537cdf20af42ce9 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=d20a7e521c7f4f93a17f7a49aa1c5d28]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:12.640784 29155 tablet_service.cc:1467] Processing CreateTablet for tablet a823ce0b7b174b53902a85b7f2e9f59f (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=d20a7e521c7f4f93a17f7a49aa1c5d28]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:12.640987 26538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0316714f34d44a5b537cdf20af42ce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:12.641187 29155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a823ce0b7b174b53902a85b7f2e9f59f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:12.653262 26798 tablet_service.cc:1467] Processing CreateTablet for tablet a823ce0b7b174b53902a85b7f2e9f59f (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=d20a7e521c7f4f93a17f7a49aa1c5d28]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:12.653788 26798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a823ce0b7b174b53902a85b7f2e9f59f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:12.654840 32542 tablet_bootstrap.cc:492] T 112821e7380e41dcac66a38e21ba809d P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:15:12.657691 32542 tablet_bootstrap.cc:654] T 112821e7380e41dcac66a38e21ba809d P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:12.657840 29553 tablet_service.cc:1467] Processing CreateTablet for tablet a823ce0b7b174b53902a85b7f2e9f59f (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=d20a7e521c7f4f93a17f7a49aa1c5d28]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:12.658318 29553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a823ce0b7b174b53902a85b7f2e9f59f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:12.676115 26278 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:15:12.687274 32548 tablet_bootstrap.cc:492] T a823ce0b7b174b53902a85b7f2e9f59f P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:15:12.688275 32542 tablet_bootstrap.cc:492] T 112821e7380e41dcac66a38e21ba809d P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:15:12.688485 32542 ts_tablet_manager.cc:1367] T 112821e7380e41dcac66a38e21ba809d P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.034s user 0.006s sys 0.010s
I20240502 14:15:12.689419 32542 raft_consensus.cc:348] T 112821e7380e41dcac66a38e21ba809d P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:12.689662 32542 raft_consensus.cc:374] T 112821e7380e41dcac66a38e21ba809d P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:12.689777 32542 raft_consensus.cc:729] T 112821e7380e41dcac66a38e21ba809d P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:15:12.690017 32542 consensus_queue.cc:260] T 112821e7380e41dcac66a38e21ba809d P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:12.690577 32542 ts_tablet_manager.cc:1398] T 112821e7380e41dcac66a38e21ba809d P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:15:12.695647 32480 tablet.cc:1722] T f46d576031a94b0d8bd30ff3dd242e19 P 1cb67158133e4648a6eceba32d4dbba3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:12.709328 32482 tablet.cc:1722] T f46d576031a94b0d8bd30ff3dd242e19 P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:12.711763 26277 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table_new_name [id=91b2a76f31334a8cbb1666ed9eb1e7f3] alter complete (version 3)
I20240502 14:15:12.712540 32493 tablet.cc:1722] T f46d576031a94b0d8bd30ff3dd242e19 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:12.741940 32548 tablet_bootstrap.cc:654] T a823ce0b7b174b53902a85b7f2e9f59f P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:12.766996 32550 tablet_bootstrap.cc:492] T a823ce0b7b174b53902a85b7f2e9f59f P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:15:12.770182 32552 tablet_bootstrap.cc:492] T b0316714f34d44a5b537cdf20af42ce9 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:15:12.772820 32552 tablet_bootstrap.cc:654] T b0316714f34d44a5b537cdf20af42ce9 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:12.775296 32550 tablet_bootstrap.cc:654] T a823ce0b7b174b53902a85b7f2e9f59f P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:12.779739 32544 tablet_bootstrap.cc:492] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:15:12.783703 32544 tablet_bootstrap.cc:654] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:12.791414 32548 tablet_bootstrap.cc:492] T a823ce0b7b174b53902a85b7f2e9f59f P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:15:12.791726 32548 ts_tablet_manager.cc:1367] T a823ce0b7b174b53902a85b7f2e9f59f P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.105s user 0.007s sys 0.004s
I20240502 14:15:12.794042 32550 tablet_bootstrap.cc:492] T a823ce0b7b174b53902a85b7f2e9f59f P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:15:12.795415 32541 tablet_bootstrap.cc:492] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:15:12.796653 32550 ts_tablet_manager.cc:1367] T a823ce0b7b174b53902a85b7f2e9f59f P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.030s user 0.001s sys 0.005s
I20240502 14:15:12.800141 32552 tablet_bootstrap.cc:492] T b0316714f34d44a5b537cdf20af42ce9 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:15:12.800359 32552 ts_tablet_manager.cc:1367] T b0316714f34d44a5b537cdf20af42ce9 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.003s
I20240502 14:15:12.801424 32552 raft_consensus.cc:348] T b0316714f34d44a5b537cdf20af42ce9 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:12.801752 32552 raft_consensus.cc:374] T b0316714f34d44a5b537cdf20af42ce9 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:12.801873 32552 raft_consensus.cc:729] T b0316714f34d44a5b537cdf20af42ce9 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:15:12.803160 32552 consensus_queue.cc:260] T b0316714f34d44a5b537cdf20af42ce9 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:12.804613 32550 raft_consensus.cc:348] T a823ce0b7b174b53902a85b7f2e9f59f P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:12.805997 32541 tablet_bootstrap.cc:654] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:12.809192 32552 ts_tablet_manager.cc:1398] T b0316714f34d44a5b537cdf20af42ce9 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 14:15:12.809585 32552 tablet_bootstrap.cc:492] T a823ce0b7b174b53902a85b7f2e9f59f P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:15:12.809746 32550 raft_consensus.cc:374] T a823ce0b7b174b53902a85b7f2e9f59f P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:12.810037 32550 raft_consensus.cc:729] T a823ce0b7b174b53902a85b7f2e9f59f P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:15:12.812327 32550 consensus_queue.cc:260] T a823ce0b7b174b53902a85b7f2e9f59f P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:12.814049 32552 tablet_bootstrap.cc:654] T a823ce0b7b174b53902a85b7f2e9f59f P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:12.819777 26281 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:12.820077 26281 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:12.824194 26281 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet f46d576031a94b0d8bd30ff3dd242e19
I20240502 14:15:12.825523 26669 tablet_service.cc:1514] Processing DeleteTablet for tablet f46d576031a94b0d8bd30ff3dd242e19 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:12 UTC) from {username='slave'} at 127.0.0.1:34150
I20240502 14:15:12.826107 32550 ts_tablet_manager.cc:1398] T a823ce0b7b174b53902a85b7f2e9f59f P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.029s user 0.002s sys 0.000s
I20240502 14:15:12.826164 32560 tablet_replica.cc:330] T f46d576031a94b0d8bd30ff3dd242e19 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: stopping tablet replica
I20240502 14:15:12.826402 32560 raft_consensus.cc:2229] T f46d576031a94b0d8bd30ff3dd242e19 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:12.826704 32560 raft_consensus.cc:2258] T f46d576031a94b0d8bd30ff3dd242e19 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:12.826730 29287 tablet_service.cc:1514] Processing DeleteTablet for tablet f46d576031a94b0d8bd30ff3dd242e19 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:12 UTC) from {username='slave'} at 127.0.0.1:59160
I20240502 14:15:12.826874 32334 raft_consensus.cc:482] T a823ce0b7b174b53902a85b7f2e9f59f P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:12.827060 32334 raft_consensus.cc:504] T a823ce0b7b174b53902a85b7f2e9f59f P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:12.827981 32560 ts_tablet_manager.cc:1875] T f46d576031a94b0d8bd30ff3dd242e19 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:12.828119 32334 leader_election.cc:288] T a823ce0b7b174b53902a85b7f2e9f59f P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:15:12.828214 29155 tablet_service.cc:1514] Processing DeleteTablet for tablet f46d576031a94b0d8bd30ff3dd242e19 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:12 UTC) from {username='slave'} at 127.0.0.1:55202
I20240502 14:15:12.828948 32562 tablet_replica.cc:330] T f46d576031a94b0d8bd30ff3dd242e19 P 1cb67158133e4648a6eceba32d4dbba3: stopping tablet replica
I20240502 14:15:12.829245 32562 raft_consensus.cc:2229] T f46d576031a94b0d8bd30ff3dd242e19 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:15:12.829241 29575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a823ce0b7b174b53902a85b7f2e9f59f" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86" is_pre_election: true
I20240502 14:15:12.829824 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a823ce0b7b174b53902a85b7f2e9f59f" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
W20240502 14:15:12.832561 29112 leader_election.cc:341] T a823ce0b7b174b53902a85b7f2e9f59f P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:12.832845 32562 raft_consensus.cc:2258] T f46d576031a94b0d8bd30ff3dd242e19 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:12.834432 32562 ts_tablet_manager.cc:1875] T f46d576031a94b0d8bd30ff3dd242e19 P 1cb67158133e4648a6eceba32d4dbba3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:12.839015 32541 tablet_bootstrap.cc:492] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:15:12.839339 32544 tablet_bootstrap.cc:492] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:15:12.839365 32541 ts_tablet_manager.cc:1367] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.044s user 0.006s sys 0.000s
I20240502 14:15:12.839490 32544 ts_tablet_manager.cc:1367] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.060s user 0.005s sys 0.000s
I20240502 14:15:12.840471 32541 raft_consensus.cc:348] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:12.840850 32541 raft_consensus.cc:374] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:12.840967 32541 raft_consensus.cc:729] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:15:12.841380 32541 consensus_queue.cc:260] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:12.842681 32541 ts_tablet_manager.cc:1398] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20240502 14:15:12.843035 29112 leader_election.cc:341] T a823ce0b7b174b53902a85b7f2e9f59f P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:12.843209 29112 leader_election.cc:302] T a823ce0b7b174b53902a85b7f2e9f59f P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3; no voters: 4092902b3e674857a0147f1b4f2cdb86, e4cfb8fc5a474177af6848a6efa13a27
I20240502 14:15:12.844043 32552 tablet_bootstrap.cc:492] T a823ce0b7b174b53902a85b7f2e9f59f P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:15:12.844333 32552 ts_tablet_manager.cc:1367] T a823ce0b7b174b53902a85b7f2e9f59f P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.003s
I20240502 14:15:12.845217 32334 raft_consensus.cc:2726] T a823ce0b7b174b53902a85b7f2e9f59f P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:15:12.847862 32548 raft_consensus.cc:348] T a823ce0b7b174b53902a85b7f2e9f59f P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:12.849571 32544 raft_consensus.cc:348] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:12.850226 32541 tablet_bootstrap.cc:492] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:15:12.845233 32552 raft_consensus.cc:348] T a823ce0b7b174b53902a85b7f2e9f59f P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:12.850571 32552 raft_consensus.cc:374] T a823ce0b7b174b53902a85b7f2e9f59f P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:12.850600 32561 tablet_replica.cc:330] T f46d576031a94b0d8bd30ff3dd242e19 P c1100e16b3b9495ca61d4f95e89b6a89: stopping tablet replica
I20240502 14:15:12.850709 32552 raft_consensus.cc:729] T a823ce0b7b174b53902a85b7f2e9f59f P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:15:12.850857 32561 raft_consensus.cc:2229] T f46d576031a94b0d8bd30ff3dd242e19 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:12.851101 32561 raft_consensus.cc:2258] T f46d576031a94b0d8bd30ff3dd242e19 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:12.851333 32548 raft_consensus.cc:374] T a823ce0b7b174b53902a85b7f2e9f59f P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:12.851487 32544 raft_consensus.cc:374] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:12.851747 32548 raft_consensus.cc:729] T a823ce0b7b174b53902a85b7f2e9f59f P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:15:12.851889 32544 raft_consensus.cc:729] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:15:12.852326 32544 consensus_queue.cc:260] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:12.852878 32552 consensus_queue.cc:260] T a823ce0b7b174b53902a85b7f2e9f59f P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:12.852934 32548 consensus_queue.cc:260] T a823ce0b7b174b53902a85b7f2e9f59f P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:12.856051 32552 ts_tablet_manager.cc:1398] T a823ce0b7b174b53902a85b7f2e9f59f P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240502 14:15:12.856632 32541 tablet_bootstrap.cc:654] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:12.861097 32544 ts_tablet_manager.cc:1398] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240502 14:15:12.862707 32548 ts_tablet_manager.cc:1398] T a823ce0b7b174b53902a85b7f2e9f59f P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.071s user 0.001s sys 0.001s
I20240502 14:15:12.863480 32561 ts_tablet_manager.cc:1875] T f46d576031a94b0d8bd30ff3dd242e19 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:12.863873 32562 ts_tablet_manager.cc:1888] T f46d576031a94b0d8bd30ff3dd242e19 P 1cb67158133e4648a6eceba32d4dbba3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:12.863914 32548 tablet_bootstrap.cc:492] T 112821e7380e41dcac66a38e21ba809d P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:15:12.866890 32562 log.cc:1196] T f46d576031a94b0d8bd30ff3dd242e19 P 1cb67158133e4648a6eceba32d4dbba3: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal/wals/f46d576031a94b0d8bd30ff3dd242e19
I20240502 14:15:12.868188 32562 ts_tablet_manager.cc:1909] T f46d576031a94b0d8bd30ff3dd242e19 P 1cb67158133e4648a6eceba32d4dbba3: Deleting consensus metadata
I20240502 14:15:12.870548 26271 catalog_manager.cc:4830] TS 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): tablet f46d576031a94b0d8bd30ff3dd242e19 (table rebalancer_dynamic_table_new_name [id=91b2a76f31334a8cbb1666ed9eb1e7f3]) successfully deleted
I20240502 14:15:12.880671 32548 tablet_bootstrap.cc:654] T 112821e7380e41dcac66a38e21ba809d P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:12.881453 32541 tablet_bootstrap.cc:492] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:15:12.881716 32541 ts_tablet_manager.cc:1367] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.032s user 0.001s sys 0.008s
I20240502 14:15:12.887516 32541 raft_consensus.cc:348] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:12.888372 32541 raft_consensus.cc:374] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:12.888535 32541 raft_consensus.cc:729] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:15:12.888865 32541 consensus_queue.cc:260] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:12.890938 32541 ts_tablet_manager.cc:1398] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240502 14:15:12.894371 32561 ts_tablet_manager.cc:1888] T f46d576031a94b0d8bd30ff3dd242e19 P c1100e16b3b9495ca61d4f95e89b6a89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:12.894524 32561 log.cc:1196] T f46d576031a94b0d8bd30ff3dd242e19 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal/wals/f46d576031a94b0d8bd30ff3dd242e19
I20240502 14:15:12.895061 32561 ts_tablet_manager.cc:1909] T f46d576031a94b0d8bd30ff3dd242e19 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting consensus metadata
I20240502 14:15:12.901659 32548 tablet_bootstrap.cc:492] T 112821e7380e41dcac66a38e21ba809d P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:15:12.901949 32548 ts_tablet_manager.cc:1367] T 112821e7380e41dcac66a38e21ba809d P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.038s user 0.002s sys 0.003s
I20240502 14:15:12.902946 32548 raft_consensus.cc:348] T 112821e7380e41dcac66a38e21ba809d P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:12.903272 32548 raft_consensus.cc:374] T 112821e7380e41dcac66a38e21ba809d P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:12.903404 32548 raft_consensus.cc:729] T 112821e7380e41dcac66a38e21ba809d P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:15:12.903764 32548 consensus_queue.cc:260] T 112821e7380e41dcac66a38e21ba809d P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:12.905836 32548 ts_tablet_manager.cc:1398] T 112821e7380e41dcac66a38e21ba809d P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 14:15:12.913074 26271 catalog_manager.cc:4830] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): tablet f46d576031a94b0d8bd30ff3dd242e19 (table rebalancer_dynamic_table_new_name [id=91b2a76f31334a8cbb1666ed9eb1e7f3]) successfully deleted
I20240502 14:15:12.926920 26276 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:15:12.939191 32560 ts_tablet_manager.cc:1888] T f46d576031a94b0d8bd30ff3dd242e19 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:12.939545 32560 log.cc:1196] T f46d576031a94b0d8bd30ff3dd242e19 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/wals/f46d576031a94b0d8bd30ff3dd242e19
I20240502 14:15:12.940915 32560 ts_tablet_manager.cc:1909] T f46d576031a94b0d8bd30ff3dd242e19 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting consensus metadata
I20240502 14:15:12.942425 26272 catalog_manager.cc:4830] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): tablet f46d576031a94b0d8bd30ff3dd242e19 (table rebalancer_dynamic_table_new_name [id=91b2a76f31334a8cbb1666ed9eb1e7f3]) successfully deleted
I20240502 14:15:12.942813 26669 tablet_service.cc:1467] Processing CreateTablet for tablet 094dc62e9e314469b1c6f9a3385fc7d6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a22fb819e98647318078753927f0a5b2]), partition=
I20240502 14:15:12.943194 26669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 094dc62e9e314469b1c6f9a3385fc7d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:12.943689 26406 tablet_service.cc:1467] Processing CreateTablet for tablet 094dc62e9e314469b1c6f9a3385fc7d6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a22fb819e98647318078753927f0a5b2]), partition=
I20240502 14:15:12.944000 26406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 094dc62e9e314469b1c6f9a3385fc7d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:12.946745 29554 tablet_service.cc:1467] Processing CreateTablet for tablet 094dc62e9e314469b1c6f9a3385fc7d6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a22fb819e98647318078753927f0a5b2]), partition=
I20240502 14:15:12.947156 29554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 094dc62e9e314469b1c6f9a3385fc7d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:12.952066 32541 tablet_bootstrap.cc:492] T 094dc62e9e314469b1c6f9a3385fc7d6 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:15:12.952277 32232 raft_consensus.cc:482] T a823ce0b7b174b53902a85b7f2e9f59f P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:12.952455 32232 raft_consensus.cc:504] T a823ce0b7b174b53902a85b7f2e9f59f P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:12.953179 32232 leader_election.cc:288] T a823ce0b7b174b53902a85b7f2e9f59f P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:15:12.953547 32541 tablet_bootstrap.cc:654] T 094dc62e9e314469b1c6f9a3385fc7d6 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:12.958592 29575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a823ce0b7b174b53902a85b7f2e9f59f" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86" is_pre_election: true
I20240502 14:15:12.958911 29575 raft_consensus.cc:2454] T a823ce0b7b174b53902a85b7f2e9f59f P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 0.
I20240502 14:15:12.959582 29176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a823ce0b7b174b53902a85b7f2e9f59f" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:15:12.959841 29176 raft_consensus.cc:2454] T a823ce0b7b174b53902a85b7f2e9f59f P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 0.
I20240502 14:15:12.960610 26754 leader_election.cc:302] T a823ce0b7b174b53902a85b7f2e9f59f P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:15:12.960983 32232 raft_consensus.cc:2781] T a823ce0b7b174b53902a85b7f2e9f59f P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:12.961063 32232 raft_consensus.cc:482] T a823ce0b7b174b53902a85b7f2e9f59f P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:12.961144 32232 raft_consensus.cc:3037] T a823ce0b7b174b53902a85b7f2e9f59f P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:12.974877 32232 raft_consensus.cc:504] T a823ce0b7b174b53902a85b7f2e9f59f P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:12.979303 32445 raft_consensus.cc:482] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:12.979455 32445 raft_consensus.cc:504] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:12.980257 32445 leader_election.cc:288] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:15:12.981194 29442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "112821e7380e41dcac66a38e21ba809d" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:15:12.981549 29442 raft_consensus.cc:2454] T 112821e7380e41dcac66a38e21ba809d P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:15:12.984568 26363 leader_election.cc:302] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:15:12.986806 32352 raft_consensus.cc:2781] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:12.986909 32352 raft_consensus.cc:482] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:12.986994 32352 raft_consensus.cc:3037] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:12.991894 32352 raft_consensus.cc:504] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:12.995862 29575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "112821e7380e41dcac66a38e21ba809d" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86" is_pre_election: true
I20240502 14:15:12.996191 29575 raft_consensus.cc:2454] T 112821e7380e41dcac66a38e21ba809d P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:15:12.997316 32352 leader_election.cc:288] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Requested vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:15:12.998919 29442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "112821e7380e41dcac66a38e21ba809d" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
I20240502 14:15:12.999122 29442 raft_consensus.cc:3037] T 112821e7380e41dcac66a38e21ba809d P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:13.007850 29442 raft_consensus.cc:2454] T 112821e7380e41dcac66a38e21ba809d P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:15:13.010154 29575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "112821e7380e41dcac66a38e21ba809d" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86"
I20240502 14:15:13.010403 29575 raft_consensus.cc:3037] T 112821e7380e41dcac66a38e21ba809d P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:13.015165 29574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a823ce0b7b174b53902a85b7f2e9f59f" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86"
I20240502 14:15:13.015456 29574 raft_consensus.cc:3037] T a823ce0b7b174b53902a85b7f2e9f59f P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:13.019726 29574 raft_consensus.cc:2454] T a823ce0b7b174b53902a85b7f2e9f59f P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
I20240502 14:15:13.021137 26755 leader_election.cc:302] T a823ce0b7b174b53902a85b7f2e9f59f P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:15:13.021567 32497 raft_consensus.cc:2781] T a823ce0b7b174b53902a85b7f2e9f59f P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:13.021735 32497 raft_consensus.cc:686] T a823ce0b7b174b53902a85b7f2e9f59f P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Becoming Leader. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Running, Role: LEADER
I20240502 14:15:13.022058 32497 consensus_queue.cc:237] T a823ce0b7b174b53902a85b7f2e9f59f P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:13.041891 32232 leader_election.cc:288] T a823ce0b7b174b53902a85b7f2e9f59f P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Requested vote from peers 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:15:13.048756 26281 catalog_manager.cc:5484] T a823ce0b7b174b53902a85b7f2e9f59f P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196). New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:13.009466 26363 leader_election.cc:302] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:15:13.052160 32445 raft_consensus.cc:2781] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:13.056098 32582 tablet_bootstrap.cc:492] T 094dc62e9e314469b1c6f9a3385fc7d6 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:15:13.063171 32582 tablet_bootstrap.cc:654] T 094dc62e9e314469b1c6f9a3385fc7d6 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:13.070606 29575 raft_consensus.cc:2454] T 112821e7380e41dcac66a38e21ba809d P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:15:13.077885 32582 tablet_bootstrap.cc:492] T 094dc62e9e314469b1c6f9a3385fc7d6 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:15:13.078181 32582 ts_tablet_manager.cc:1367] T 094dc62e9e314469b1c6f9a3385fc7d6 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.008s
I20240502 14:15:13.080062 32582 raft_consensus.cc:348] T 094dc62e9e314469b1c6f9a3385fc7d6 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:13.080490 32582 raft_consensus.cc:374] T 094dc62e9e314469b1c6f9a3385fc7d6 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:13.080761 32582 raft_consensus.cc:729] T 094dc62e9e314469b1c6f9a3385fc7d6 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:15:13.064576 32587 raft_consensus.cc:482] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:13.081090 32582 consensus_queue.cc:260] T 094dc62e9e314469b1c6f9a3385fc7d6 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:13.081214 32587 raft_consensus.cc:504] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:13.084367 32587 leader_election.cc:288] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:15:13.085104 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0316714f34d44a5b537cdf20af42ce9" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:15:13.085352 26558 raft_consensus.cc:2454] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:15:13.085915 26362 leader_election.cc:302] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:15:13.087343 32587 raft_consensus.cc:2781] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:13.087518 32587 raft_consensus.cc:482] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:13.087685 32587 raft_consensus.cc:3037] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:13.088873 32582 ts_tablet_manager.cc:1398] T 094dc62e9e314469b1c6f9a3385fc7d6 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240502 14:15:13.090243 32548 tablet_bootstrap.cc:492] T 094dc62e9e314469b1c6f9a3385fc7d6 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:15:13.092610 32548 tablet_bootstrap.cc:654] T 094dc62e9e314469b1c6f9a3385fc7d6 P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:13.106552 32587 raft_consensus.cc:504] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:13.108605 32548 tablet_bootstrap.cc:492] T 094dc62e9e314469b1c6f9a3385fc7d6 P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:15:13.109319 32548 ts_tablet_manager.cc:1367] T 094dc62e9e314469b1c6f9a3385fc7d6 P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.004s
I20240502 14:15:13.110399 32548 raft_consensus.cc:348] T 094dc62e9e314469b1c6f9a3385fc7d6 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:13.124783 29421 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:15:13.133471 32548 raft_consensus.cc:374] T 094dc62e9e314469b1c6f9a3385fc7d6 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:13.133880 32548 raft_consensus.cc:729] T 094dc62e9e314469b1c6f9a3385fc7d6 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:15:13.134474 32548 consensus_queue.cc:260] T 094dc62e9e314469b1c6f9a3385fc7d6 P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:13.134832 26669 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:15:13.139266 32548 ts_tablet_manager.cc:1398] T 094dc62e9e314469b1c6f9a3385fc7d6 P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.030s user 0.002s sys 0.000s
I20240502 14:15:13.139765 29176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a823ce0b7b174b53902a85b7f2e9f59f" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:15:13.140033 29176 raft_consensus.cc:3037] T a823ce0b7b174b53902a85b7f2e9f59f P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:13.146494 29176 raft_consensus.cc:2454] T a823ce0b7b174b53902a85b7f2e9f59f P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
I20240502 14:15:13.180644 32497 raft_consensus.cc:482] T b0316714f34d44a5b537cdf20af42ce9 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:13.180845 32497 raft_consensus.cc:504] T b0316714f34d44a5b537cdf20af42ce9 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:13.181674 32497 leader_election.cc:288] T b0316714f34d44a5b537cdf20af42ce9 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:15:13.182999 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0316714f34d44a5b537cdf20af42ce9" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:15:13.183274 26558 raft_consensus.cc:2454] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 0.
I20240502 14:15:13.183727 26754 leader_election.cc:302] T b0316714f34d44a5b537cdf20af42ce9 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:15:13.184242 32497 raft_consensus.cc:2781] T b0316714f34d44a5b537cdf20af42ce9 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:13.184331 32497 raft_consensus.cc:482] T b0316714f34d44a5b537cdf20af42ce9 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:13.184417 32497 raft_consensus.cc:3037] T b0316714f34d44a5b537cdf20af42ce9 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:13.190209 32497 raft_consensus.cc:504] T b0316714f34d44a5b537cdf20af42ce9 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:13.193276 29287 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:13.193977 32497 leader_election.cc:288] T b0316714f34d44a5b537cdf20af42ce9 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Requested vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
W20240502 14:15:13.111920 26363 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.199:41781, user_credentials={real_user=slave}} blocked reactor thread for 58690.8us
I20240502 14:15:13.117738 32541 tablet_bootstrap.cc:492] T 094dc62e9e314469b1c6f9a3385fc7d6 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:15:13.196056 32541 ts_tablet_manager.cc:1367] T 094dc62e9e314469b1c6f9a3385fc7d6 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.244s user 0.008s sys 0.004s
I20240502 14:15:13.121194 32445 raft_consensus.cc:686] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Becoming Leader. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Running, Role: LEADER
I20240502 14:15:13.183001 26423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0316714f34d44a5b537cdf20af42ce9" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:15:13.191819 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0316714f34d44a5b537cdf20af42ce9" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:15:13.196671 32445 consensus_queue.cc:237] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:13.196951 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0316714f34d44a5b537cdf20af42ce9" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:15:13.197170 26818 raft_consensus.cc:2379] T b0316714f34d44a5b537cdf20af42ce9 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 89af7f423cf44717a08d6b142f0ed49d in current term 1: Already voted for candidate e4cfb8fc5a474177af6848a6efa13a27 in this term.
I20240502 14:15:13.198289 32541 raft_consensus.cc:348] T 094dc62e9e314469b1c6f9a3385fc7d6 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:13.198541 32541 raft_consensus.cc:374] T 094dc62e9e314469b1c6f9a3385fc7d6 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:13.198602 32541 raft_consensus.cc:729] T 094dc62e9e314469b1c6f9a3385fc7d6 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:15:13.198846 32541 consensus_queue.cc:260] T 094dc62e9e314469b1c6f9a3385fc7d6 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:13.199540 32541 ts_tablet_manager.cc:1398] T 094dc62e9e314469b1c6f9a3385fc7d6 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:15:13.213330 29155 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:13.214553 26423 raft_consensus.cc:2379] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4cfb8fc5a474177af6848a6efa13a27 in current term 1: Already voted for candidate 89af7f423cf44717a08d6b142f0ed49d in this term.
I20240502 14:15:13.215864 32587 leader_election.cc:288] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Requested vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:15:13.216509 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0316714f34d44a5b537cdf20af42ce9" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:15:13.216799 26558 raft_consensus.cc:3037] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:13.228631 26555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0316714f34d44a5b537cdf20af42ce9" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:15:13.230170 26754 leader_election.cc:302] T b0316714f34d44a5b537cdf20af42ce9 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4cfb8fc5a474177af6848a6efa13a27; no voters: 89af7f423cf44717a08d6b142f0ed49d, ba187e38a2754d60a12de9ac4856430a
I20240502 14:15:13.230463 32232 raft_consensus.cc:2726] T b0316714f34d44a5b537cdf20af42ce9 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 14:15:13.232124 26558 raft_consensus.cc:2454] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:15:13.233424 26362 leader_election.cc:302] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:15:13.238266 32584 raft_consensus.cc:2781] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:13.238437 32584 raft_consensus.cc:686] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Becoming Leader. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Running, Role: LEADER
I20240502 14:15:13.238700 32584 consensus_queue.cc:237] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:13.232867 26278 catalog_manager.cc:5484] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: term changed from 0 to 1, leader changed from <none> to 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193). New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:15:13.240607 26819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0316714f34d44a5b537cdf20af42ce9" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:15:13.240864 26819 raft_consensus.cc:2379] T b0316714f34d44a5b537cdf20af42ce9 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 89af7f423cf44717a08d6b142f0ed49d in current term 1: Already voted for candidate e4cfb8fc5a474177af6848a6efa13a27 in this term.
I20240502 14:15:13.246506 26281 catalog_manager.cc:5484] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: term changed from 0 to 1, leader changed from <none> to 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193). New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:13.276759 26538 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:15:13.334553 29554 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:15:13.346208 32605 raft_consensus.cc:482] T 094dc62e9e314469b1c6f9a3385fc7d6 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:13.346431 32605 raft_consensus.cc:504] T 094dc62e9e314469b1c6f9a3385fc7d6 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:13.346881 32375 raft_consensus.cc:482] T 094dc62e9e314469b1c6f9a3385fc7d6 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:13.347162 32605 leader_election.cc:288] T 094dc62e9e314469b1c6f9a3385fc7d6 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:15:13.347097 32375 raft_consensus.cc:504] T 094dc62e9e314469b1c6f9a3385fc7d6 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:13.347781 32375 leader_election.cc:288] T 094dc62e9e314469b1c6f9a3385fc7d6 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:15:13.349480 26689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "094dc62e9e314469b1c6f9a3385fc7d6" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:15:13.349700 26689 raft_consensus.cc:2454] T 094dc62e9e314469b1c6f9a3385fc7d6 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:15:13.351091 29574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "094dc62e9e314469b1c6f9a3385fc7d6" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86" is_pre_election: true
I20240502 14:15:13.351774 29511 leader_election.cc:302] T 094dc62e9e314469b1c6f9a3385fc7d6 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters:
I20240502 14:15:13.351990 29574 raft_consensus.cc:2454] T 094dc62e9e314469b1c6f9a3385fc7d6 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:15:13.352196 32375 raft_consensus.cc:2781] T 094dc62e9e314469b1c6f9a3385fc7d6 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:13.352300 32375 raft_consensus.cc:482] T 094dc62e9e314469b1c6f9a3385fc7d6 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:13.352423 26625 leader_election.cc:302] T 094dc62e9e314469b1c6f9a3385fc7d6 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters:
I20240502 14:15:13.352741 32605 raft_consensus.cc:2781] T 094dc62e9e314469b1c6f9a3385fc7d6 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:13.352847 32605 raft_consensus.cc:482] T 094dc62e9e314469b1c6f9a3385fc7d6 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:13.352929 32605 raft_consensus.cc:3037] T 094dc62e9e314469b1c6f9a3385fc7d6 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:13.353276 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "094dc62e9e314469b1c6f9a3385fc7d6" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:15:13.353307 26423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "094dc62e9e314469b1c6f9a3385fc7d6" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:15:13.353492 26426 raft_consensus.cc:2454] T 094dc62e9e314469b1c6f9a3385fc7d6 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:15:13.353657 32375 raft_consensus.cc:3037] T 094dc62e9e314469b1c6f9a3385fc7d6 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:13.356009 32605 raft_consensus.cc:504] T 094dc62e9e314469b1c6f9a3385fc7d6 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:13.359262 29574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "094dc62e9e314469b1c6f9a3385fc7d6" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86"
I20240502 14:15:13.359822 32605 leader_election.cc:288] T 094dc62e9e314469b1c6f9a3385fc7d6 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Requested vote from peers 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:15:13.360285 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "094dc62e9e314469b1c6f9a3385fc7d6" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:15:13.360522 26426 raft_consensus.cc:3037] T 094dc62e9e314469b1c6f9a3385fc7d6 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:13.361459 32375 raft_consensus.cc:504] T 094dc62e9e314469b1c6f9a3385fc7d6 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:13.363487 26426 raft_consensus.cc:2454] T 094dc62e9e314469b1c6f9a3385fc7d6 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:15:13.365535 26624 leader_election.cc:302] T 094dc62e9e314469b1c6f9a3385fc7d6 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:15:13.362037 29574 raft_consensus.cc:2379] T 094dc62e9e314469b1c6f9a3385fc7d6 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in current term 1: Already voted for candidate 4092902b3e674857a0147f1b4f2cdb86 in this term.
I20240502 14:15:13.366668 32370 raft_consensus.cc:2781] T 094dc62e9e314469b1c6f9a3385fc7d6 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:13.366825 32370 raft_consensus.cc:686] T 094dc62e9e314469b1c6f9a3385fc7d6 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Becoming Leader. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Running, Role: LEADER
I20240502 14:15:13.367096 32370 consensus_queue.cc:237] T 094dc62e9e314469b1c6f9a3385fc7d6 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:13.371217 26687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "094dc62e9e314469b1c6f9a3385fc7d6" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:15:13.374219 26423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "094dc62e9e314469b1c6f9a3385fc7d6" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:15:13.374526 26423 raft_consensus.cc:2379] T 094dc62e9e314469b1c6f9a3385fc7d6 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4092902b3e674857a0147f1b4f2cdb86 in current term 1: Already voted for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in this term.
I20240502 14:15:13.375108 29510 leader_election.cc:302] T 094dc62e9e314469b1c6f9a3385fc7d6 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86; no voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, 89af7f423cf44717a08d6b142f0ed49d
I20240502 14:15:13.375455 31623 raft_consensus.cc:2726] T 094dc62e9e314469b1c6f9a3385fc7d6 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 14:15:13.375829 32375 leader_election.cc:288] T 094dc62e9e314469b1c6f9a3385fc7d6 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Requested vote from peers 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:15:13.385619 26406 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:13.412895 26281 catalog_manager.cc:5484] T 094dc62e9e314469b1c6f9a3385fc7d6 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195). New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:13.424944 26797 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:15:13.435103 26624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.94.193:37985, user_credentials={real_user=slave}} blocked reactor thread for 69592.4us
I20240502 14:15:13.518838 32232 consensus_queue.cc:1035] T a823ce0b7b174b53902a85b7f2e9f59f P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:13.552187 26281 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:15:13.579705 32232 consensus_queue.cc:1035] T a823ce0b7b174b53902a85b7f2e9f59f P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:13.593115 32445 consensus_queue.cc:1035] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:13.594594 29575 raft_consensus.cc:1261] T 094dc62e9e314469b1c6f9a3385fc7d6 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:13.595851 32370 consensus_queue.cc:1035] T 094dc62e9e314469b1c6f9a3385fc7d6 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:13.598425 26426 raft_consensus.cc:1261] T 094dc62e9e314469b1c6f9a3385fc7d6 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:13.599529 32370 consensus_queue.cc:1035] T 094dc62e9e314469b1c6f9a3385fc7d6 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:13.635224 26281 catalog_manager.cc:5576] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985) does not have the latest schema for tablet 094dc62e9e314469b1c6f9a3385fc7d6 (table rebalancer_dynamic_table [id=a22fb819e98647318078753927f0a5b2]). Expected version 1 got 0
I20240502 14:15:13.662173 26281 catalog_manager.cc:5576] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097) does not have the latest schema for tablet 094dc62e9e314469b1c6f9a3385fc7d6 (table rebalancer_dynamic_table [id=a22fb819e98647318078753927f0a5b2]). Expected version 1 got 0
I20240502 14:15:13.676934 32445 consensus_queue.cc:1035] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:13.684576 32620 tablet.cc:1722] T 094dc62e9e314469b1c6f9a3385fc7d6 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:13.685593 32627 tablet.cc:1722] T 094dc62e9e314469b1c6f9a3385fc7d6 P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:13.687652 32620 tablet.cc:1620] T 094dc62e9e314469b1c6f9a3385fc7d6 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: MemRowSet was empty: no flush needed.
I20240502 14:15:13.688161 32627 tablet.cc:1620] T 094dc62e9e314469b1c6f9a3385fc7d6 P 4092902b3e674857a0147f1b4f2cdb86: MemRowSet was empty: no flush needed.
I20240502 14:15:13.694006 26276 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=a22fb819e98647318078753927f0a5b2] alter complete (version 1)
I20240502 14:15:13.704447 32621 tablet.cc:1722] T 094dc62e9e314469b1c6f9a3385fc7d6 P 89af7f423cf44717a08d6b142f0ed49d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:13.705256 32621 tablet.cc:1620] T 094dc62e9e314469b1c6f9a3385fc7d6 P 89af7f423cf44717a08d6b142f0ed49d: MemRowSet was empty: no flush needed.
I20240502 14:15:13.717746 26281 catalog_manager.cc:5576] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985) does not have the latest schema for tablet 094dc62e9e314469b1c6f9a3385fc7d6 (table rebalancer_dynamic_table [id=a22fb819e98647318078753927f0a5b2]). Expected version 1 got 0
I20240502 14:15:13.763358 32585 consensus_queue.cc:1035] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:13.781307 32584 consensus_queue.cc:1035] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:13.782910 26281 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_extra_table_46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:13.805219 26276 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:15:13.818401 29287 tablet_service.cc:1467] Processing CreateTablet for tablet a511c4f2b4c14f4a90872bf0093c035c (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=cf2fa69bdc0c4ba4af966938f6e97bd7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:13.819001 29287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a511c4f2b4c14f4a90872bf0093c035c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:13.820030 29421 tablet_service.cc:1467] Processing CreateTablet for tablet a511c4f2b4c14f4a90872bf0093c035c (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=cf2fa69bdc0c4ba4af966938f6e97bd7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:13.820497 29421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a511c4f2b4c14f4a90872bf0093c035c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:13.816967 26538 tablet_service.cc:1467] Processing CreateTablet for tablet a511c4f2b4c14f4a90872bf0093c035c (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=cf2fa69bdc0c4ba4af966938f6e97bd7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:13.821581 26538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a511c4f2b4c14f4a90872bf0093c035c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:13.826907 32620 tablet.cc:1722] T 094dc62e9e314469b1c6f9a3385fc7d6 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:13.827474 32620 tablet.cc:1620] T 094dc62e9e314469b1c6f9a3385fc7d6 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: MemRowSet was empty: no flush needed.
I20240502 14:15:13.828912 32627 tablet.cc:1722] T 094dc62e9e314469b1c6f9a3385fc7d6 P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:13.829417 32627 tablet.cc:1620] T 094dc62e9e314469b1c6f9a3385fc7d6 P 4092902b3e674857a0147f1b4f2cdb86: MemRowSet was empty: no flush needed.
I20240502 14:15:13.838186 32639 tablet_bootstrap.cc:492] T a511c4f2b4c14f4a90872bf0093c035c P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:15:13.840781 32639 tablet_bootstrap.cc:654] T a511c4f2b4c14f4a90872bf0093c035c P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:13.844696 32621 tablet.cc:1722] T 094dc62e9e314469b1c6f9a3385fc7d6 P 89af7f423cf44717a08d6b142f0ed49d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:13.845219 32621 tablet.cc:1620] T 094dc62e9e314469b1c6f9a3385fc7d6 P 89af7f423cf44717a08d6b142f0ed49d: MemRowSet was empty: no flush needed.
I20240502 14:15:13.862111 26276 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=a22fb819e98647318078753927f0a5b2] alter complete (version 2)
I20240502 14:15:13.868309 32639 tablet_bootstrap.cc:492] T a511c4f2b4c14f4a90872bf0093c035c P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:15:13.868738 32639 ts_tablet_manager.cc:1367] T a511c4f2b4c14f4a90872bf0093c035c P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.031s user 0.010s sys 0.001s
I20240502 14:15:13.872308 32639 raft_consensus.cc:348] T a511c4f2b4c14f4a90872bf0093c035c P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:13.872829 32639 raft_consensus.cc:374] T a511c4f2b4c14f4a90872bf0093c035c P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:13.873184 32639 raft_consensus.cc:729] T a511c4f2b4c14f4a90872bf0093c035c P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:15:13.873644 32639 consensus_queue.cc:260] T a511c4f2b4c14f4a90872bf0093c035c P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:13.874470 26669 tablet_service.cc:1467] Processing CreateTablet for tablet e665b144dfe541dfaf492281d2f72100 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=cf2fa69bdc0c4ba4af966938f6e97bd7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:13.874971 26669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e665b144dfe541dfaf492281d2f72100. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:13.875058 32639 ts_tablet_manager.cc:1398] T a511c4f2b4c14f4a90872bf0093c035c P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 14:15:13.898448 29422 tablet_service.cc:1467] Processing CreateTablet for tablet e665b144dfe541dfaf492281d2f72100 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=cf2fa69bdc0c4ba4af966938f6e97bd7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:13.899052 29422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e665b144dfe541dfaf492281d2f72100. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:13.906625 26668 tablet_service.cc:1467] Processing CreateTablet for tablet 220adc4ff7dc4a3b957e479da22fbd3d (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=cf2fa69bdc0c4ba4af966938f6e97bd7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:13.907176 26668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 220adc4ff7dc4a3b957e479da22fbd3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:13.926729 32646 tablet_bootstrap.cc:492] T e665b144dfe541dfaf492281d2f72100 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:15:13.929679 32644 tablet_bootstrap.cc:492] T a511c4f2b4c14f4a90872bf0093c035c P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:15:13.932334 26537 tablet_service.cc:1467] Processing CreateTablet for tablet e665b144dfe541dfaf492281d2f72100 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=cf2fa69bdc0c4ba4af966938f6e97bd7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:13.932955 26537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e665b144dfe541dfaf492281d2f72100. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:13.934616 29155 tablet_service.cc:1467] Processing CreateTablet for tablet 220adc4ff7dc4a3b957e479da22fbd3d (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=cf2fa69bdc0c4ba4af966938f6e97bd7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:13.934922 26406 tablet_service.cc:1467] Processing CreateTablet for tablet 220adc4ff7dc4a3b957e479da22fbd3d (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=cf2fa69bdc0c4ba4af966938f6e97bd7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:13.935089 29155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 220adc4ff7dc4a3b957e479da22fbd3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:13.935232 32643 tablet_bootstrap.cc:492] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:15:13.935464 26406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 220adc4ff7dc4a3b957e479da22fbd3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:13.936733 32644 tablet_bootstrap.cc:654] T a511c4f2b4c14f4a90872bf0093c035c P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:13.947355 32643 tablet_bootstrap.cc:654] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:13.928686 32646 tablet_bootstrap.cc:654] T e665b144dfe541dfaf492281d2f72100 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:13.949780 32652 tablet_bootstrap.cc:492] T 220adc4ff7dc4a3b957e479da22fbd3d P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
W20240502 14:15:13.955245 26493 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.197:38715, user_credentials={real_user=slave}} blocked reactor thread for 47388.1us
I20240502 14:15:13.956358 32646 tablet_bootstrap.cc:492] T e665b144dfe541dfaf492281d2f72100 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:15:13.956590 32646 ts_tablet_manager.cc:1367] T e665b144dfe541dfaf492281d2f72100 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.001s
I20240502 14:15:13.956866 32652 tablet_bootstrap.cc:654] T 220adc4ff7dc4a3b957e479da22fbd3d P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:13.957729 32646 raft_consensus.cc:348] T e665b144dfe541dfaf492281d2f72100 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:13.958094 32646 raft_consensus.cc:374] T e665b144dfe541dfaf492281d2f72100 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:13.958222 32646 raft_consensus.cc:729] T e665b144dfe541dfaf492281d2f72100 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:15:13.958690 32646 consensus_queue.cc:260] T e665b144dfe541dfaf492281d2f72100 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:13.959545 32646 ts_tablet_manager.cc:1398] T e665b144dfe541dfaf492281d2f72100 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:15:13.961084 32646 tablet_bootstrap.cc:492] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:15:13.963479 32646 tablet_bootstrap.cc:654] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:13.970353 32652 tablet_bootstrap.cc:492] T 220adc4ff7dc4a3b957e479da22fbd3d P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:15:13.970530 32652 ts_tablet_manager.cc:1367] T 220adc4ff7dc4a3b957e479da22fbd3d P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.003s
I20240502 14:15:13.972235 32652 raft_consensus.cc:348] T 220adc4ff7dc4a3b957e479da22fbd3d P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:13.973069 26278 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:15:13.983224 32652 raft_consensus.cc:374] T 220adc4ff7dc4a3b957e479da22fbd3d P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:13.983536 32652 raft_consensus.cc:729] T 220adc4ff7dc4a3b957e479da22fbd3d P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:15:13.984117 32652 consensus_queue.cc:260] T 220adc4ff7dc4a3b957e479da22fbd3d P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:13.985814 32652 ts_tablet_manager.cc:1398] T 220adc4ff7dc4a3b957e479da22fbd3d P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.015s user 0.000s sys 0.002s
I20240502 14:15:14.000803 32657 tablet_bootstrap.cc:492] T 220adc4ff7dc4a3b957e479da22fbd3d P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:15:14.001906 32646 tablet_bootstrap.cc:492] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:15:14.002188 32646 ts_tablet_manager.cc:1367] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.041s user 0.008s sys 0.003s
I20240502 14:15:14.005033 32620 tablet.cc:1722] T 094dc62e9e314469b1c6f9a3385fc7d6 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:14.005220 32657 tablet_bootstrap.cc:654] T 220adc4ff7dc4a3b957e479da22fbd3d P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:14.006431 32621 tablet.cc:1722] T 094dc62e9e314469b1c6f9a3385fc7d6 P 89af7f423cf44717a08d6b142f0ed49d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:14.012050 32627 tablet.cc:1722] T 094dc62e9e314469b1c6f9a3385fc7d6 P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:14.012957 32646 raft_consensus.cc:348] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:14.013258 32646 raft_consensus.cc:374] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:14.016201 32646 raft_consensus.cc:729] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:15:14.018136 26281 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table_new_name [id=a22fb819e98647318078753927f0a5b2] alter complete (version 3)
I20240502 14:15:14.017113 32646 consensus_queue.cc:260] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:14.020763 32646 ts_tablet_manager.cc:1398] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.018s user 0.002s sys 0.001s
I20240502 14:15:14.028203 32643 tablet_bootstrap.cc:492] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:15:14.028378 32643 ts_tablet_manager.cc:1367] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.093s user 0.004s sys 0.019s
I20240502 14:15:14.029217 32643 raft_consensus.cc:348] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:14.029457 32643 raft_consensus.cc:374] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:14.029522 32643 raft_consensus.cc:729] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:15:14.029729 32643 consensus_queue.cc:260] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:14.030294 32644 tablet_bootstrap.cc:492] T a511c4f2b4c14f4a90872bf0093c035c P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:15:14.030299 32643 ts_tablet_manager.cc:1398] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:15:14.030586 32643 tablet_bootstrap.cc:492] T e665b144dfe541dfaf492281d2f72100 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:15:14.030581 32644 ts_tablet_manager.cc:1367] T a511c4f2b4c14f4a90872bf0093c035c P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.101s user 0.027s sys 0.004s
I20240502 14:15:14.032150 32643 tablet_bootstrap.cc:654] T e665b144dfe541dfaf492281d2f72100 P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:14.031469 32644 raft_consensus.cc:348] T a511c4f2b4c14f4a90872bf0093c035c P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:14.032971 32644 raft_consensus.cc:374] T a511c4f2b4c14f4a90872bf0093c035c P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:14.033066 32644 raft_consensus.cc:729] T a511c4f2b4c14f4a90872bf0093c035c P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:15:14.033332 32644 consensus_queue.cc:260] T a511c4f2b4c14f4a90872bf0093c035c P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:14.040369 32644 ts_tablet_manager.cc:1398] T a511c4f2b4c14f4a90872bf0093c035c P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240502 14:15:14.046105 32644 tablet_bootstrap.cc:492] T e665b144dfe541dfaf492281d2f72100 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:15:14.050343 32644 tablet_bootstrap.cc:654] T e665b144dfe541dfaf492281d2f72100 P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:14.050673 32643 tablet_bootstrap.cc:492] T e665b144dfe541dfaf492281d2f72100 P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:15:14.050940 32643 ts_tablet_manager.cc:1367] T e665b144dfe541dfaf492281d2f72100 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.006s
I20240502 14:15:14.051919 32643 raft_consensus.cc:348] T e665b144dfe541dfaf492281d2f72100 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:14.052222 32643 raft_consensus.cc:374] T e665b144dfe541dfaf492281d2f72100 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:14.053560 32643 raft_consensus.cc:729] T e665b144dfe541dfaf492281d2f72100 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:15:14.053900 32643 consensus_queue.cc:260] T e665b144dfe541dfaf492281d2f72100 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:14.054107 32657 tablet_bootstrap.cc:492] T 220adc4ff7dc4a3b957e479da22fbd3d P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:15:14.054313 32657 ts_tablet_manager.cc:1367] T 220adc4ff7dc4a3b957e479da22fbd3d P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.054s user 0.005s sys 0.002s
I20240502 14:15:14.056895 32657 raft_consensus.cc:348] T 220adc4ff7dc4a3b957e479da22fbd3d P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:14.057293 32657 raft_consensus.cc:374] T 220adc4ff7dc4a3b957e479da22fbd3d P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:14.057530 32657 raft_consensus.cc:729] T 220adc4ff7dc4a3b957e479da22fbd3d P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:15:14.057909 32657 consensus_queue.cc:260] T 220adc4ff7dc4a3b957e479da22fbd3d P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:14.059453 32643 ts_tablet_manager.cc:1398] T e665b144dfe541dfaf492281d2f72100 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240502 14:15:14.064642 32657 ts_tablet_manager.cc:1398] T 220adc4ff7dc4a3b957e479da22fbd3d P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 14:15:14.065663 32644 tablet_bootstrap.cc:492] T e665b144dfe541dfaf492281d2f72100 P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:15:14.065881 32644 ts_tablet_manager.cc:1367] T e665b144dfe541dfaf492281d2f72100 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.004s
I20240502 14:15:14.066799 32644 raft_consensus.cc:348] T e665b144dfe541dfaf492281d2f72100 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:14.067086 32644 raft_consensus.cc:374] T e665b144dfe541dfaf492281d2f72100 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:14.067198 32644 raft_consensus.cc:729] T e665b144dfe541dfaf492281d2f72100 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:15:14.067497 32644 consensus_queue.cc:260] T e665b144dfe541dfaf492281d2f72100 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:14.068171 32644 ts_tablet_manager.cc:1398] T e665b144dfe541dfaf492281d2f72100 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:15:14.094203 26558 consensus_queue.cc:237] T 825ae3e90e214a6f98394765edf6ff2d P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:14.097352 26423 raft_consensus.cc:1261] T 825ae3e90e214a6f98394765edf6ff2d P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:14.097491 29574 raft_consensus.cc:1261] T 825ae3e90e214a6f98394765edf6ff2d P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:14.098089 32650 consensus_queue.cc:1035] T 825ae3e90e214a6f98394765edf6ff2d P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:14.098371 31369 consensus_queue.cc:1035] T 825ae3e90e214a6f98394765edf6ff2d P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:14.101753 31369 raft_consensus.cc:2932] T 825ae3e90e214a6f98394765edf6ff2d P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:15:14.103871 29574 raft_consensus.cc:2932] T 825ae3e90e214a6f98394765edf6ff2d P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:15:14.107224 32263 raft_consensus.cc:482] T a511c4f2b4c14f4a90872bf0093c035c P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:14.107407 32263 raft_consensus.cc:504] T a511c4f2b4c14f4a90872bf0093c035c P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:14.108227 32263 leader_election.cc:288] T a511c4f2b4c14f4a90872bf0093c035c P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:15:14.109079 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a511c4f2b4c14f4a90872bf0093c035c" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:15:14.109249 26558 raft_consensus.cc:2454] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 0.
I20240502 14:15:14.109728 29304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a511c4f2b4c14f4a90872bf0093c035c" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:15:14.110000 29304 raft_consensus.cc:2454] T a511c4f2b4c14f4a90872bf0093c035c P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 0.
I20240502 14:15:14.111529 29378 leader_election.cc:302] T a511c4f2b4c14f4a90872bf0093c035c P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:15:14.112730 26276 catalog_manager.cc:5484] T 825ae3e90e214a6f98394765edf6ff2d P ba187e38a2754d60a12de9ac4856430a reported cstate change: config changed from index 4 to 5, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:15:14.113382 32263 raft_consensus.cc:2781] T a511c4f2b4c14f4a90872bf0093c035c P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:14.113519 32263 raft_consensus.cc:482] T a511c4f2b4c14f4a90872bf0093c035c P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:14.113603 32263 raft_consensus.cc:3037] T a511c4f2b4c14f4a90872bf0093c035c P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:14.116842 32263 raft_consensus.cc:504] T a511c4f2b4c14f4a90872bf0093c035c P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:14.117457 32263 leader_election.cc:288] T a511c4f2b4c14f4a90872bf0093c035c P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 election: Requested vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:15:14.117995 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a511c4f2b4c14f4a90872bf0093c035c" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:15:14.118196 26558 raft_consensus.cc:3037] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:14.121114 29304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a511c4f2b4c14f4a90872bf0093c035c" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
I20240502 14:15:14.121366 29304 raft_consensus.cc:3037] T a511c4f2b4c14f4a90872bf0093c035c P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:14.123550 29304 raft_consensus.cc:2454] T a511c4f2b4c14f4a90872bf0093c035c P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 1.
I20240502 14:15:14.125370 26284 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:14.125550 26284 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:14.127923 32608 raft_consensus.cc:482] T e665b144dfe541dfaf492281d2f72100 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:14.128125 32608 raft_consensus.cc:504] T e665b144dfe541dfaf492281d2f72100 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:14.132150 32608 leader_election.cc:288] T e665b144dfe541dfaf492281d2f72100 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:15:14.132284 26284 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet 094dc62e9e314469b1c6f9a3385fc7d6
I20240502 14:15:14.133572 26406 tablet_service.cc:1514] Processing DeleteTablet for tablet 094dc62e9e314469b1c6f9a3385fc7d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:14 UTC) from {username='slave'} at 127.0.0.1:37422
I20240502 14:15:14.135011 29554 tablet_service.cc:1514] Processing DeleteTablet for tablet 094dc62e9e314469b1c6f9a3385fc7d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:14 UTC) from {username='slave'} at 127.0.0.1:36398
I20240502 14:15:14.135165 26669 tablet_service.cc:1514] Processing DeleteTablet for tablet 094dc62e9e314469b1c6f9a3385fc7d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:14 UTC) from {username='slave'} at 127.0.0.1:34150
I20240502 14:15:14.133685 26555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e665b144dfe541dfaf492281d2f72100" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:15:14.135630 26555 raft_consensus.cc:2454] T e665b144dfe541dfaf492281d2f72100 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:15:14.138917 26624 leader_election.cc:302] T e665b144dfe541dfaf492281d2f72100 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:15:14.139267 32608 raft_consensus.cc:2781] T e665b144dfe541dfaf492281d2f72100 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:14.139369 32608 raft_consensus.cc:482] T e665b144dfe541dfaf492281d2f72100 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:14.139465 32608 raft_consensus.cc:3037] T e665b144dfe541dfaf492281d2f72100 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:14.140821 26558 raft_consensus.cc:2454] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 1.
I20240502 14:15:14.147418 26797 tablet_service.cc:1514] Processing DeleteTablet for tablet 825ae3e90e214a6f98394765edf6ff2d with delete_type TABLET_DATA_TOMBSTONED (TS e4cfb8fc5a474177af6848a6efa13a27 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:56010
I20240502 14:15:14.149659 26423 raft_consensus.cc:2932] T 825ae3e90e214a6f98394765edf6ff2d P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:15:14.124487 29378 leader_election.cc:302] T a511c4f2b4c14f4a90872bf0093c035c P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:15:14.133157 29442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e665b144dfe541dfaf492281d2f72100" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:15:14.150990 29442 raft_consensus.cc:2454] T e665b144dfe541dfaf492281d2f72100 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:15:14.156701 32263 raft_consensus.cc:2781] T a511c4f2b4c14f4a90872bf0093c035c P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:14.158366 32263 raft_consensus.cc:686] T a511c4f2b4c14f4a90872bf0093c035c P 68a3c2b17ed94e06aba66ed4178132ab [term 1 LEADER]: Becoming Leader. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Running, Role: LEADER
I20240502 14:15:14.150072 32608 raft_consensus.cc:504] T e665b144dfe541dfaf492281d2f72100 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:14.167088 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e665b144dfe541dfaf492281d2f72100" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:15:14.167392 26558 raft_consensus.cc:3037] T e665b144dfe541dfaf492281d2f72100 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:14.168254 29442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e665b144dfe541dfaf492281d2f72100" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
I20240502 14:15:14.168550 29442 raft_consensus.cc:3037] T e665b144dfe541dfaf492281d2f72100 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:14.170329 26558 raft_consensus.cc:2454] T e665b144dfe541dfaf492281d2f72100 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:15:14.170761 26624 leader_election.cc:302] T e665b144dfe541dfaf492281d2f72100 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:15:14.171073 32370 raft_consensus.cc:2781] T e665b144dfe541dfaf492281d2f72100 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:14.171394 29442 raft_consensus.cc:2454] T e665b144dfe541dfaf492281d2f72100 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:15:14.171805 32672 tablet_replica.cc:330] T 094dc62e9e314469b1c6f9a3385fc7d6 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: stopping tablet replica
I20240502 14:15:14.172034 32672 raft_consensus.cc:2229] T 094dc62e9e314469b1c6f9a3385fc7d6 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:15:14.172624 32672 raft_consensus.cc:2258] T 094dc62e9e314469b1c6f9a3385fc7d6 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:14.173862 32672 ts_tablet_manager.cc:1875] T 094dc62e9e314469b1c6f9a3385fc7d6 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:14.174944 32263 consensus_queue.cc:237] T a511c4f2b4c14f4a90872bf0093c035c P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:14.184852 32608 leader_election.cc:288] T e665b144dfe541dfaf492281d2f72100 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Requested vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:15:14.185767 32608 raft_consensus.cc:482] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:14.186906 32608 raft_consensus.cc:504] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:14.189137 26423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220adc4ff7dc4a3b957e479da22fbd3d" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:15:14.189368 26423 raft_consensus.cc:2454] T 220adc4ff7dc4a3b957e479da22fbd3d P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:15:14.189826 26624 leader_election.cc:302] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:15:14.190788 29176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220adc4ff7dc4a3b957e479da22fbd3d" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:15:14.191063 29176 raft_consensus.cc:2454] T 220adc4ff7dc4a3b957e479da22fbd3d P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:15:14.191948 32608 leader_election.cc:288] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:15:14.192649 32608 raft_consensus.cc:2781] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:14.194620 32370 raft_consensus.cc:686] T e665b144dfe541dfaf492281d2f72100 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Becoming Leader. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Running, Role: LEADER
I20240502 14:15:14.195935 32608 raft_consensus.cc:482] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:14.196118 32608 raft_consensus.cc:3037] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:14.196153 32370 consensus_queue.cc:237] T e665b144dfe541dfaf492281d2f72100 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:14.206792 32675 tablet_replica.cc:330] T 825ae3e90e214a6f98394765edf6ff2d P e4cfb8fc5a474177af6848a6efa13a27: stopping tablet replica
I20240502 14:15:14.207101 32675 raft_consensus.cc:2229] T 825ae3e90e214a6f98394765edf6ff2d P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:14.207358 32675 raft_consensus.cc:2258] T 825ae3e90e214a6f98394765edf6ff2d P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:14.208289 32675 ts_tablet_manager.cc:1875] T 825ae3e90e214a6f98394765edf6ff2d P e4cfb8fc5a474177af6848a6efa13a27: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:14.212679 32671 tablet_replica.cc:330] T 094dc62e9e314469b1c6f9a3385fc7d6 P 4092902b3e674857a0147f1b4f2cdb86: stopping tablet replica
I20240502 14:15:14.213110 32671 raft_consensus.cc:2229] T 094dc62e9e314469b1c6f9a3385fc7d6 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:14.213471 32671 raft_consensus.cc:2258] T 094dc62e9e314469b1c6f9a3385fc7d6 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:14.216580 32672 ts_tablet_manager.cc:1888] T 094dc62e9e314469b1c6f9a3385fc7d6 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:14.219723 32672 log.cc:1196] T 094dc62e9e314469b1c6f9a3385fc7d6 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/wals/094dc62e9e314469b1c6f9a3385fc7d6
W20240502 14:15:14.221408 29378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.94.198:43659, user_credentials={real_user=slave}} blocked reactor thread for 96898.5us
I20240502 14:15:14.223340 32670 tablet_replica.cc:330] T 094dc62e9e314469b1c6f9a3385fc7d6 P 89af7f423cf44717a08d6b142f0ed49d: stopping tablet replica
I20240502 14:15:14.225929 32670 raft_consensus.cc:2229] T 094dc62e9e314469b1c6f9a3385fc7d6 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:14.225934 26281 catalog_manager.cc:5484] T a511c4f2b4c14f4a90872bf0093c035c P 68a3c2b17ed94e06aba66ed4178132ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199). New cstate: current_term: 1 leader_uuid: "68a3c2b17ed94e06aba66ed4178132ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:15:14.226312 32608 raft_consensus.cc:504] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:14.226928 32670 raft_consensus.cc:2258] T 094dc62e9e314469b1c6f9a3385fc7d6 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:14.226840 26276 catalog_manager.cc:5484] T e665b144dfe541dfaf492281d2f72100 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195). New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:15:14.228040 32672 ts_tablet_manager.cc:1909] T 094dc62e9e314469b1c6f9a3385fc7d6 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting consensus metadata
I20240502 14:15:14.228067 26423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220adc4ff7dc4a3b957e479da22fbd3d" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:15:14.228271 26423 raft_consensus.cc:3037] T 220adc4ff7dc4a3b957e479da22fbd3d P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:14.228790 32671 ts_tablet_manager.cc:1875] T 094dc62e9e314469b1c6f9a3385fc7d6 P 4092902b3e674857a0147f1b4f2cdb86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:14.229310 26272 catalog_manager.cc:4830] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): tablet 094dc62e9e314469b1c6f9a3385fc7d6 (table rebalancer_dynamic_table_new_name [id=a22fb819e98647318078753927f0a5b2]) successfully deleted
I20240502 14:15:14.229431 32608 leader_election.cc:288] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Requested vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:15:14.230486 29176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220adc4ff7dc4a3b957e479da22fbd3d" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:15:14.230724 29176 raft_consensus.cc:3037] T 220adc4ff7dc4a3b957e479da22fbd3d P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:14.231117 26423 raft_consensus.cc:2454] T 220adc4ff7dc4a3b957e479da22fbd3d P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:15:14.232200 26624 leader_election.cc:302] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:15:14.232539 32608 raft_consensus.cc:2781] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:14.232776 32608 raft_consensus.cc:686] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Becoming Leader. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Running, Role: LEADER
I20240502 14:15:14.233088 32608 consensus_queue.cc:237] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:14.233577 29176 raft_consensus.cc:2454] T 220adc4ff7dc4a3b957e479da22fbd3d P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:15:14.237360 32675 ts_tablet_manager.cc:1888] T 825ae3e90e214a6f98394765edf6ff2d P e4cfb8fc5a474177af6848a6efa13a27: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:15:14.237720 32675 log.cc:1196] T 825ae3e90e214a6f98394765edf6ff2d P e4cfb8fc5a474177af6848a6efa13a27: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/wals/825ae3e90e214a6f98394765edf6ff2d
I20240502 14:15:14.238715 26272 catalog_manager.cc:4830] TS e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): tablet 825ae3e90e214a6f98394765edf6ff2d (table rebalance_test_table_9 [id=3c3eeaa25da94395a642aa569b652348]) successfully deleted
I20240502 14:15:14.245565 26281 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:15:14.249158 32671 ts_tablet_manager.cc:1888] T 094dc62e9e314469b1c6f9a3385fc7d6 P 4092902b3e674857a0147f1b4f2cdb86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:14.249420 32671 log.cc:1196] T 094dc62e9e314469b1c6f9a3385fc7d6 P 4092902b3e674857a0147f1b4f2cdb86: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/wal/wals/094dc62e9e314469b1c6f9a3385fc7d6
I20240502 14:15:14.250020 32671 ts_tablet_manager.cc:1909] T 094dc62e9e314469b1c6f9a3385fc7d6 P 4092902b3e674857a0147f1b4f2cdb86: Deleting consensus metadata
I20240502 14:15:14.250015 32670 ts_tablet_manager.cc:1875] T 094dc62e9e314469b1c6f9a3385fc7d6 P 89af7f423cf44717a08d6b142f0ed49d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:14.251276 26272 catalog_manager.cc:4830] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097): tablet 094dc62e9e314469b1c6f9a3385fc7d6 (table rebalancer_dynamic_table_new_name [id=a22fb819e98647318078753927f0a5b2]) successfully deleted
I20240502 14:15:14.254611 26276 catalog_manager.cc:5484] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195). New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:14.265657 29287 tablet_service.cc:1467] Processing CreateTablet for tablet 6220cb9484c943bf9a56cd9401ea7acf (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9a19a749ad534538b0ad1503e7563faa]), partition=
I20240502 14:15:14.265892 29155 tablet_service.cc:1467] Processing CreateTablet for tablet 6220cb9484c943bf9a56cd9401ea7acf (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9a19a749ad534538b0ad1503e7563faa]), partition=
I20240502 14:15:14.266172 29287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6220cb9484c943bf9a56cd9401ea7acf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:14.266227 29155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6220cb9484c943bf9a56cd9401ea7acf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:14.266402 29554 tablet_service.cc:1467] Processing CreateTablet for tablet 6220cb9484c943bf9a56cd9401ea7acf (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9a19a749ad534538b0ad1503e7563faa]), partition=
I20240502 14:15:14.266824 29554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6220cb9484c943bf9a56cd9401ea7acf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:14.273571 32657 tablet_bootstrap.cc:492] T 6220cb9484c943bf9a56cd9401ea7acf P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:15:14.275400 32657 tablet_bootstrap.cc:654] T 6220cb9484c943bf9a56cd9401ea7acf P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:14.275507 32686 tablet_bootstrap.cc:492] T 6220cb9484c943bf9a56cd9401ea7acf P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:15:14.276546 32670 ts_tablet_manager.cc:1888] T 094dc62e9e314469b1c6f9a3385fc7d6 P 89af7f423cf44717a08d6b142f0ed49d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:14.277388 32686 tablet_bootstrap.cc:654] T 6220cb9484c943bf9a56cd9401ea7acf P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:14.277747 32670 log.cc:1196] T 094dc62e9e314469b1c6f9a3385fc7d6 P 89af7f423cf44717a08d6b142f0ed49d: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/094dc62e9e314469b1c6f9a3385fc7d6
I20240502 14:15:14.278898 32670 ts_tablet_manager.cc:1909] T 094dc62e9e314469b1c6f9a3385fc7d6 P 89af7f423cf44717a08d6b142f0ed49d: Deleting consensus metadata
I20240502 14:15:14.280748 32639 tablet_bootstrap.cc:492] T 6220cb9484c943bf9a56cd9401ea7acf P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:15:14.282930 32639 tablet_bootstrap.cc:654] T 6220cb9484c943bf9a56cd9401ea7acf P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:14.289817 26271 catalog_manager.cc:4830] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): tablet 094dc62e9e314469b1c6f9a3385fc7d6 (table rebalancer_dynamic_table_new_name [id=a22fb819e98647318078753927f0a5b2]) successfully deleted
I20240502 14:15:14.294142 32686 tablet_bootstrap.cc:492] T 6220cb9484c943bf9a56cd9401ea7acf P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:15:14.294432 32686 ts_tablet_manager.cc:1367] T 6220cb9484c943bf9a56cd9401ea7acf P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.010s
I20240502 14:15:14.295312 32657 tablet_bootstrap.cc:492] T 6220cb9484c943bf9a56cd9401ea7acf P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:15:14.295513 32686 raft_consensus.cc:348] T 6220cb9484c943bf9a56cd9401ea7acf P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:14.295997 32686 raft_consensus.cc:374] T 6220cb9484c943bf9a56cd9401ea7acf P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:14.296166 32686 raft_consensus.cc:729] T 6220cb9484c943bf9a56cd9401ea7acf P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:15:14.296589 32686 consensus_queue.cc:260] T 6220cb9484c943bf9a56cd9401ea7acf P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:14.298352 32657 ts_tablet_manager.cc:1367] T 6220cb9484c943bf9a56cd9401ea7acf P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.001s
I20240502 14:15:14.299830 32657 raft_consensus.cc:348] T 6220cb9484c943bf9a56cd9401ea7acf P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:14.300175 32639 tablet_bootstrap.cc:492] T 6220cb9484c943bf9a56cd9401ea7acf P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:15:14.300372 32657 raft_consensus.cc:374] T 6220cb9484c943bf9a56cd9401ea7acf P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:14.300599 32657 raft_consensus.cc:729] T 6220cb9484c943bf9a56cd9401ea7acf P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:15:14.301014 32657 consensus_queue.cc:260] T 6220cb9484c943bf9a56cd9401ea7acf P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:14.300364 32639 ts_tablet_manager.cc:1367] T 6220cb9484c943bf9a56cd9401ea7acf P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.006s
I20240502 14:15:14.302136 31623 raft_consensus.cc:482] T 6220cb9484c943bf9a56cd9401ea7acf P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:14.302361 31623 raft_consensus.cc:504] T 6220cb9484c943bf9a56cd9401ea7acf P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:14.302994 32639 raft_consensus.cc:348] T 6220cb9484c943bf9a56cd9401ea7acf P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:14.303121 31623 leader_election.cc:288] T 6220cb9484c943bf9a56cd9401ea7acf P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:15:14.303225 32639 raft_consensus.cc:374] T 6220cb9484c943bf9a56cd9401ea7acf P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:14.303310 32639 raft_consensus.cc:729] T 6220cb9484c943bf9a56cd9401ea7acf P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:15:14.303562 32639 consensus_queue.cc:260] T 6220cb9484c943bf9a56cd9401ea7acf P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:14.306025 29175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6220cb9484c943bf9a56cd9401ea7acf" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:15:14.306283 29175 raft_consensus.cc:2454] T 6220cb9484c943bf9a56cd9401ea7acf P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:15:14.306773 32639 ts_tablet_manager.cc:1398] T 6220cb9484c943bf9a56cd9401ea7acf P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:15:14.306912 29304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6220cb9484c943bf9a56cd9401ea7acf" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:15:14.307183 29304 raft_consensus.cc:2454] T 6220cb9484c943bf9a56cd9401ea7acf P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:15:14.308709 32657 ts_tablet_manager.cc:1398] T 6220cb9484c943bf9a56cd9401ea7acf P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.010s user 0.000s sys 0.003s
I20240502 14:15:14.314345 29510 leader_election.cc:302] T 6220cb9484c943bf9a56cd9401ea7acf P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 4092902b3e674857a0147f1b4f2cdb86; no voters:
I20240502 14:15:14.314970 32607 raft_consensus.cc:2781] T 6220cb9484c943bf9a56cd9401ea7acf P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:14.315085 32607 raft_consensus.cc:482] T 6220cb9484c943bf9a56cd9401ea7acf P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:14.315201 32607 raft_consensus.cc:3037] T 6220cb9484c943bf9a56cd9401ea7acf P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:14.318135 32607 raft_consensus.cc:504] T 6220cb9484c943bf9a56cd9401ea7acf P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:14.318780 32607 leader_election.cc:288] T 6220cb9484c943bf9a56cd9401ea7acf P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Requested vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:15:14.320566 29175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6220cb9484c943bf9a56cd9401ea7acf" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:15:14.320832 29175 raft_consensus.cc:3037] T 6220cb9484c943bf9a56cd9401ea7acf P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:14.325117 32686 ts_tablet_manager.cc:1398] T 6220cb9484c943bf9a56cd9401ea7acf P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.030s user 0.000s sys 0.002s
I20240502 14:15:14.325590 29304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6220cb9484c943bf9a56cd9401ea7acf" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
I20240502 14:15:14.325829 29304 raft_consensus.cc:3037] T 6220cb9484c943bf9a56cd9401ea7acf P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:14.328938 29304 raft_consensus.cc:2454] T 6220cb9484c943bf9a56cd9401ea7acf P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:15:14.330765 29175 raft_consensus.cc:2454] T 6220cb9484c943bf9a56cd9401ea7acf P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:15:14.331013 29510 leader_election.cc:302] T 6220cb9484c943bf9a56cd9401ea7acf P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:15:14.331375 31623 raft_consensus.cc:2781] T 6220cb9484c943bf9a56cd9401ea7acf P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:14.331528 31623 raft_consensus.cc:686] T 6220cb9484c943bf9a56cd9401ea7acf P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Becoming Leader. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Running, Role: LEADER
I20240502 14:15:14.331811 31623 consensus_queue.cc:237] T 6220cb9484c943bf9a56cd9401ea7acf P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:14.357220 26276 catalog_manager.cc:5484] T 6220cb9484c943bf9a56cd9401ea7acf P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200). New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:14.464100 26276 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:15:14.472839 29175 raft_consensus.cc:1261] T 6220cb9484c943bf9a56cd9401ea7acf P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:14.473670 32696 consensus_queue.cc:1035] T 6220cb9484c943bf9a56cd9401ea7acf P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:14.474062 29304 raft_consensus.cc:1261] T 6220cb9484c943bf9a56cd9401ea7acf P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:14.478763 31623 consensus_queue.cc:1035] T 6220cb9484c943bf9a56cd9401ea7acf P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:14.491209 26284 catalog_manager.cc:5576] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659) does not have the latest schema for tablet 6220cb9484c943bf9a56cd9401ea7acf (table rebalancer_dynamic_table [id=9a19a749ad534538b0ad1503e7563faa]). Expected version 1 got 0
I20240502 14:15:14.493053 32627 tablet.cc:1722] T 6220cb9484c943bf9a56cd9401ea7acf P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:14.493355 26284 catalog_manager.cc:5576] TS 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715) does not have the latest schema for tablet 6220cb9484c943bf9a56cd9401ea7acf (table rebalancer_dynamic_table [id=9a19a749ad534538b0ad1503e7563faa]). Expected version 1 got 0
I20240502 14:15:14.493613 32627 tablet.cc:1620] T 6220cb9484c943bf9a56cd9401ea7acf P 4092902b3e674857a0147f1b4f2cdb86: MemRowSet was empty: no flush needed.
I20240502 14:15:14.495398 32719 tablet.cc:1722] T 6220cb9484c943bf9a56cd9401ea7acf P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:14.496380 32719 tablet.cc:1620] T 6220cb9484c943bf9a56cd9401ea7acf P c1100e16b3b9495ca61d4f95e89b6a89: MemRowSet was empty: no flush needed.
I20240502 14:15:14.499395 26284 catalog_manager.cc:5576] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659) does not have the latest schema for tablet 6220cb9484c943bf9a56cd9401ea7acf (table rebalancer_dynamic_table [id=9a19a749ad534538b0ad1503e7563faa]). Expected version 1 got 0
I20240502 14:15:14.500803 32720 tablet.cc:1722] T 6220cb9484c943bf9a56cd9401ea7acf P 1cb67158133e4648a6eceba32d4dbba3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:14.502456 32720 tablet.cc:1620] T 6220cb9484c943bf9a56cd9401ea7acf P 1cb67158133e4648a6eceba32d4dbba3: MemRowSet was empty: no flush needed.
I20240502 14:15:14.505753 26276 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=9a19a749ad534538b0ad1503e7563faa] alter complete (version 1)
I20240502 14:15:14.508028 26276 catalog_manager.cc:5576] TS 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715) does not have the latest schema for tablet 6220cb9484c943bf9a56cd9401ea7acf (table rebalancer_dynamic_table [id=9a19a749ad534538b0ad1503e7563faa]). Expected version 1 got 0
I20240502 14:15:14.608141 26284 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:15:14.608635 32676 consensus_queue.cc:1035] T a511c4f2b4c14f4a90872bf0093c035c P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:14.615875 32676 consensus_queue.cc:1035] T a511c4f2b4c14f4a90872bf0093c035c P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:14.621286 32627 tablet.cc:1722] T 6220cb9484c943bf9a56cd9401ea7acf P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:14.621886 32627 tablet.cc:1620] T 6220cb9484c943bf9a56cd9401ea7acf P 4092902b3e674857a0147f1b4f2cdb86: MemRowSet was empty: no flush needed.
I20240502 14:15:14.639117 32720 tablet.cc:1722] T 6220cb9484c943bf9a56cd9401ea7acf P 1cb67158133e4648a6eceba32d4dbba3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:14.639876 32720 tablet.cc:1620] T 6220cb9484c943bf9a56cd9401ea7acf P 1cb67158133e4648a6eceba32d4dbba3: MemRowSet was empty: no flush needed.
I20240502 14:15:14.659134 32719 tablet.cc:1722] T 6220cb9484c943bf9a56cd9401ea7acf P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:14.659680 32719 tablet.cc:1620] T 6220cb9484c943bf9a56cd9401ea7acf P c1100e16b3b9495ca61d4f95e89b6a89: MemRowSet was empty: no flush needed.
I20240502 14:15:14.688135 32608 consensus_queue.cc:1035] T e665b144dfe541dfaf492281d2f72100 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:14.711730 32608 consensus_queue.cc:1035] T e665b144dfe541dfaf492281d2f72100 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:14.718611 26276 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=9a19a749ad534538b0ad1503e7563faa] alter complete (version 2)
I20240502 14:15:14.721395 32678 consensus_queue.cc:1035] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:14.732208 32608 consensus_queue.cc:1035] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:14.776965 26284 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_extra_table_48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:14.815335 26669 tablet_service.cc:1467] Processing CreateTablet for tablet 0f2a4042e0f64ceebb345b9cf9764805 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=53d47c43f2f14c30b29a5147461aafae]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:14.815809 26669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f2a4042e0f64ceebb345b9cf9764805. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:14.830964 26406 tablet_service.cc:1467] Processing CreateTablet for tablet 0f2a4042e0f64ceebb345b9cf9764805 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=53d47c43f2f14c30b29a5147461aafae]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:14.832171 26797 tablet_service.cc:1467] Processing CreateTablet for tablet 0f2a4042e0f64ceebb345b9cf9764805 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=53d47c43f2f14c30b29a5147461aafae]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:14.832670 26797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f2a4042e0f64ceebb345b9cf9764805. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:14.833122 26406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f2a4042e0f64ceebb345b9cf9764805. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:14.835614 26798 tablet_service.cc:1467] Processing CreateTablet for tablet e45de4b8415446a08e9550f911367624 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=53d47c43f2f14c30b29a5147461aafae]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:14.836112 26798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e45de4b8415446a08e9550f911367624. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:14.836618 26407 tablet_service.cc:1467] Processing CreateTablet for tablet e45de4b8415446a08e9550f911367624 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=53d47c43f2f14c30b29a5147461aafae]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:14.836714 29288 tablet_service.cc:1467] Processing CreateTablet for tablet 2dc3b47b4ab145ccb9efbf1b5740c7f6 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=53d47c43f2f14c30b29a5147461aafae]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:14.837121 29288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dc3b47b4ab145ccb9efbf1b5740c7f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:14.840824 29287 tablet_service.cc:1467] Processing CreateTablet for tablet e45de4b8415446a08e9550f911367624 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=53d47c43f2f14c30b29a5147461aafae]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:14.840828 26405 tablet_service.cc:1467] Processing CreateTablet for tablet 2dc3b47b4ab145ccb9efbf1b5740c7f6 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=53d47c43f2f14c30b29a5147461aafae]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:14.841250 29287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e45de4b8415446a08e9550f911367624. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:14.841346 26281 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:15:14.842129 26407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e45de4b8415446a08e9550f911367624. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:14.842351 26405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dc3b47b4ab145ccb9efbf1b5740c7f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:14.845341 29554 tablet_service.cc:1467] Processing CreateTablet for tablet 2dc3b47b4ab145ccb9efbf1b5740c7f6 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=53d47c43f2f14c30b29a5147461aafae]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:14.845794 29554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dc3b47b4ab145ccb9efbf1b5740c7f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:14.854568 32627 tablet.cc:1722] T 6220cb9484c943bf9a56cd9401ea7acf P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:14.855428 32720 tablet.cc:1722] T 6220cb9484c943bf9a56cd9401ea7acf P 1cb67158133e4648a6eceba32d4dbba3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:14.864742 26281 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table_new_name [id=9a19a749ad534538b0ad1503e7563faa] alter complete (version 3)
I20240502 14:15:14.865079 32719 tablet.cc:1722] T 6220cb9484c943bf9a56cd9401ea7acf P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:14.872689 32748 tablet_bootstrap.cc:492] T 0f2a4042e0f64ceebb345b9cf9764805 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:15:14.879513 32750 tablet_bootstrap.cc:492] T e45de4b8415446a08e9550f911367624 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:15:14.883599 32752 tablet_bootstrap.cc:492] T 0f2a4042e0f64ceebb345b9cf9764805 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:15:14.885600 32750 tablet_bootstrap.cc:654] T e45de4b8415446a08e9550f911367624 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:14.886255 32751 tablet_bootstrap.cc:492] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:15:14.888588 32748 tablet_bootstrap.cc:654] T 0f2a4042e0f64ceebb345b9cf9764805 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:14.889729 32752 tablet_bootstrap.cc:654] T 0f2a4042e0f64ceebb345b9cf9764805 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:14.902726 32748 tablet_bootstrap.cc:492] T 0f2a4042e0f64ceebb345b9cf9764805 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:15:14.902963 32748 ts_tablet_manager.cc:1367] T 0f2a4042e0f64ceebb345b9cf9764805 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.030s user 0.001s sys 0.004s
I20240502 14:15:14.903672 32748 raft_consensus.cc:348] T 0f2a4042e0f64ceebb345b9cf9764805 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:14.903981 32748 raft_consensus.cc:374] T 0f2a4042e0f64ceebb345b9cf9764805 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:14.904076 32748 raft_consensus.cc:729] T 0f2a4042e0f64ceebb345b9cf9764805 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:15:14.904361 32748 consensus_queue.cc:260] T 0f2a4042e0f64ceebb345b9cf9764805 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:14.908663 32751 tablet_bootstrap.cc:654] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:14.909030 32748 ts_tablet_manager.cc:1398] T 0f2a4042e0f64ceebb345b9cf9764805 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 14:15:14.910351 32749 tablet_bootstrap.cc:492] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:15:14.921901 32752 tablet_bootstrap.cc:492] T 0f2a4042e0f64ceebb345b9cf9764805 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:15:14.925570 32750 tablet_bootstrap.cc:492] T e45de4b8415446a08e9550f911367624 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:15:14.925959 32750 ts_tablet_manager.cc:1367] T e45de4b8415446a08e9550f911367624 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.047s user 0.004s sys 0.000s
I20240502 14:15:14.926250 32752 ts_tablet_manager.cc:1367] T 0f2a4042e0f64ceebb345b9cf9764805 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.043s user 0.004s sys 0.000s
I20240502 14:15:14.928105 32749 tablet_bootstrap.cc:654] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:14.929772 32750 raft_consensus.cc:348] T e45de4b8415446a08e9550f911367624 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:14.931448 32752 raft_consensus.cc:348] T 0f2a4042e0f64ceebb345b9cf9764805 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:14.931670 32752 raft_consensus.cc:374] T 0f2a4042e0f64ceebb345b9cf9764805 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:14.932566 32751 tablet_bootstrap.cc:492] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:15:14.932691 32750 raft_consensus.cc:374] T e45de4b8415446a08e9550f911367624 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:14.932943 32752 raft_consensus.cc:729] T 0f2a4042e0f64ceebb345b9cf9764805 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:15:14.933161 32750 raft_consensus.cc:729] T e45de4b8415446a08e9550f911367624 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:15:14.933255 32751 ts_tablet_manager.cc:1367] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.047s user 0.001s sys 0.003s
I20240502 14:15:14.933595 32750 consensus_queue.cc:260] T e45de4b8415446a08e9550f911367624 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:14.933982 32752 consensus_queue.cc:260] T 0f2a4042e0f64ceebb345b9cf9764805 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:14.934178 32751 raft_consensus.cc:348] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:14.934392 32751 raft_consensus.cc:374] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:14.934502 32751 raft_consensus.cc:729] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:15:14.934744 32751 consensus_queue.cc:260] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:14.935496 32751 ts_tablet_manager.cc:1398] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:15:14.941737 32749 tablet_bootstrap.cc:492] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:15:14.942409 32752 ts_tablet_manager.cc:1398] T 0f2a4042e0f64ceebb345b9cf9764805 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240502 14:15:14.944933 32750 ts_tablet_manager.cc:1398] T e45de4b8415446a08e9550f911367624 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20240502 14:15:14.945402 32749 ts_tablet_manager.cc:1367] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.000s
I20240502 14:15:14.945834 32752 tablet_bootstrap.cc:492] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:15:14.949139 32750 tablet_bootstrap.cc:492] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:15:14.950878 32752 tablet_bootstrap.cc:654] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:14.952701 32750 tablet_bootstrap.cc:654] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:14.954190 32749 raft_consensus.cc:348] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:14.957165 32749 raft_consensus.cc:374] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:14.957481 32749 raft_consensus.cc:729] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:15:14.958032 32749 consensus_queue.cc:260] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:14.964493 32749 ts_tablet_manager.cc:1398] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240502 14:15:14.971585 26284 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:14.971849 26284 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:14.973212 32749 tablet_bootstrap.cc:492] T e45de4b8415446a08e9550f911367624 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:15:14.973729 32752 tablet_bootstrap.cc:492] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:15:14.975708 32750 tablet_bootstrap.cc:492] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:15:14.977607 32752 ts_tablet_manager.cc:1367] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.000s
I20240502 14:15:14.980233 32752 raft_consensus.cc:348] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:14.980751 32750 ts_tablet_manager.cc:1367] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.000s
I20240502 14:15:14.981930 26284 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet 6220cb9484c943bf9a56cd9401ea7acf
I20240502 14:15:14.983009 32749 tablet_bootstrap.cc:654] T e45de4b8415446a08e9550f911367624 P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:14.983572 32752 raft_consensus.cc:374] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:14.983875 32752 raft_consensus.cc:729] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:15:14.982000 32750 raft_consensus.cc:348] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:14.985064 29554 tablet_service.cc:1514] Processing DeleteTablet for tablet 6220cb9484c943bf9a56cd9401ea7acf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:14 UTC) from {username='slave'} at 127.0.0.1:36398
I20240502 14:15:14.985786 29155 tablet_service.cc:1514] Processing DeleteTablet for tablet 6220cb9484c943bf9a56cd9401ea7acf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:14 UTC) from {username='slave'} at 127.0.0.1:55202
I20240502 14:15:14.988549 32752 consensus_queue.cc:260] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:14.988904 32750 raft_consensus.cc:374] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:14.990552 32767 tablet_replica.cc:330] T 6220cb9484c943bf9a56cd9401ea7acf P 4092902b3e674857a0147f1b4f2cdb86: stopping tablet replica
I20240502 14:15:14.984159 29288 tablet_service.cc:1514] Processing DeleteTablet for tablet 6220cb9484c943bf9a56cd9401ea7acf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:14 UTC) from {username='slave'} at 127.0.0.1:59160
I20240502 14:15:14.991492 300 tablet_replica.cc:330] T 6220cb9484c943bf9a56cd9401ea7acf P 1cb67158133e4648a6eceba32d4dbba3: stopping tablet replica
I20240502 14:15:14.991829 300 raft_consensus.cc:2229] T 6220cb9484c943bf9a56cd9401ea7acf P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:14.992127 300 raft_consensus.cc:2258] T 6220cb9484c943bf9a56cd9401ea7acf P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:14.992357 32750 raft_consensus.cc:729] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:15:14.993136 32752 ts_tablet_manager.cc:1398] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
I20240502 14:15:14.993336 300 ts_tablet_manager.cc:1875] T 6220cb9484c943bf9a56cd9401ea7acf P 1cb67158133e4648a6eceba32d4dbba3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:14.993278 32750 consensus_queue.cc:260] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:14.994580 32750 ts_tablet_manager.cc:1398] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240502 14:15:15.000756 300 ts_tablet_manager.cc:1888] T 6220cb9484c943bf9a56cd9401ea7acf P 1cb67158133e4648a6eceba32d4dbba3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:15.000958 300 log.cc:1196] T 6220cb9484c943bf9a56cd9401ea7acf P 1cb67158133e4648a6eceba32d4dbba3: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal/wals/6220cb9484c943bf9a56cd9401ea7acf
I20240502 14:15:15.001605 300 ts_tablet_manager.cc:1909] T 6220cb9484c943bf9a56cd9401ea7acf P 1cb67158133e4648a6eceba32d4dbba3: Deleting consensus metadata
I20240502 14:15:15.002202 32750 tablet_bootstrap.cc:492] T 0f2a4042e0f64ceebb345b9cf9764805 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:15:15.003001 26271 catalog_manager.cc:4830] TS 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): tablet 6220cb9484c943bf9a56cd9401ea7acf (table rebalancer_dynamic_table_new_name [id=9a19a749ad534538b0ad1503e7563faa]) successfully deleted
I20240502 14:15:15.005383 32750 tablet_bootstrap.cc:654] T 0f2a4042e0f64ceebb345b9cf9764805 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:15.009667 32750 tablet_bootstrap.cc:492] T 0f2a4042e0f64ceebb345b9cf9764805 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:15:15.009862 32750 ts_tablet_manager.cc:1367] T 0f2a4042e0f64ceebb345b9cf9764805 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240502 14:15:15.010512 32750 raft_consensus.cc:348] T 0f2a4042e0f64ceebb345b9cf9764805 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:15.010684 32750 raft_consensus.cc:374] T 0f2a4042e0f64ceebb345b9cf9764805 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:15.010797 32750 raft_consensus.cc:729] T 0f2a4042e0f64ceebb345b9cf9764805 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:15:15.011102 32750 consensus_queue.cc:260] T 0f2a4042e0f64ceebb345b9cf9764805 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:15.013406 32750 ts_tablet_manager.cc:1398] T 0f2a4042e0f64ceebb345b9cf9764805 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:15:15.016218 32767 raft_consensus.cc:2229] T 6220cb9484c943bf9a56cd9401ea7acf P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:15:15.019658 32749 tablet_bootstrap.cc:492] T e45de4b8415446a08e9550f911367624 P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:15:15.019873 32749 ts_tablet_manager.cc:1367] T e45de4b8415446a08e9550f911367624 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.049s user 0.007s sys 0.007s
I20240502 14:15:15.020876 32749 raft_consensus.cc:348] T e45de4b8415446a08e9550f911367624 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:15.021200 32749 raft_consensus.cc:374] T e45de4b8415446a08e9550f911367624 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:15.021328 32749 raft_consensus.cc:729] T e45de4b8415446a08e9550f911367624 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:15:15.021652 32749 consensus_queue.cc:260] T e45de4b8415446a08e9550f911367624 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:15.016760 32767 raft_consensus.cc:2258] T 6220cb9484c943bf9a56cd9401ea7acf P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:15.023918 32749 ts_tablet_manager.cc:1398] T e45de4b8415446a08e9550f911367624 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:15:15.024803 32767 ts_tablet_manager.cc:1875] T 6220cb9484c943bf9a56cd9401ea7acf P 4092902b3e674857a0147f1b4f2cdb86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:15.041723 32767 ts_tablet_manager.cc:1888] T 6220cb9484c943bf9a56cd9401ea7acf P 4092902b3e674857a0147f1b4f2cdb86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:15.042317 32767 log.cc:1196] T 6220cb9484c943bf9a56cd9401ea7acf P 4092902b3e674857a0147f1b4f2cdb86: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/wal/wals/6220cb9484c943bf9a56cd9401ea7acf
I20240502 14:15:15.043203 32767 ts_tablet_manager.cc:1909] T 6220cb9484c943bf9a56cd9401ea7acf P 4092902b3e674857a0147f1b4f2cdb86: Deleting consensus metadata
I20240502 14:15:15.051802 26272 catalog_manager.cc:4830] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097): tablet 6220cb9484c943bf9a56cd9401ea7acf (table rebalancer_dynamic_table_new_name [id=9a19a749ad534538b0ad1503e7563faa]) successfully deleted
I20240502 14:15:15.053633 301 tablet_replica.cc:330] T 6220cb9484c943bf9a56cd9401ea7acf P c1100e16b3b9495ca61d4f95e89b6a89: stopping tablet replica
I20240502 14:15:15.053856 301 raft_consensus.cc:2229] T 6220cb9484c943bf9a56cd9401ea7acf P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:15.054271 301 raft_consensus.cc:2258] T 6220cb9484c943bf9a56cd9401ea7acf P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:15.055585 301 ts_tablet_manager.cc:1875] T 6220cb9484c943bf9a56cd9401ea7acf P c1100e16b3b9495ca61d4f95e89b6a89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:15.063763 301 ts_tablet_manager.cc:1888] T 6220cb9484c943bf9a56cd9401ea7acf P c1100e16b3b9495ca61d4f95e89b6a89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:15.064014 301 log.cc:1196] T 6220cb9484c943bf9a56cd9401ea7acf P c1100e16b3b9495ca61d4f95e89b6a89: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal/wals/6220cb9484c943bf9a56cd9401ea7acf
I20240502 14:15:15.064601 301 ts_tablet_manager.cc:1909] T 6220cb9484c943bf9a56cd9401ea7acf P c1100e16b3b9495ca61d4f95e89b6a89: Deleting consensus metadata
I20240502 14:15:15.065847 26271 catalog_manager.cc:4830] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): tablet 6220cb9484c943bf9a56cd9401ea7acf (table rebalancer_dynamic_table_new_name [id=9a19a749ad534538b0ad1503e7563faa]) successfully deleted
I20240502 14:15:15.074748 32678 raft_consensus.cc:482] T 0f2a4042e0f64ceebb345b9cf9764805 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:15.074913 32678 raft_consensus.cc:504] T 0f2a4042e0f64ceebb345b9cf9764805 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:15.075579 32678 leader_election.cc:288] T 0f2a4042e0f64ceebb345b9cf9764805 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:15:15.076695 26423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f2a4042e0f64ceebb345b9cf9764805" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:15:15.076923 26423 raft_consensus.cc:2454] T 0f2a4042e0f64ceebb345b9cf9764805 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:15:15.077519 26624 leader_election.cc:302] T 0f2a4042e0f64ceebb345b9cf9764805 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:15:15.080850 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f2a4042e0f64ceebb345b9cf9764805" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:15:15.081127 26818 raft_consensus.cc:2454] T 0f2a4042e0f64ceebb345b9cf9764805 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 0.
I20240502 14:15:15.082180 32677 raft_consensus.cc:2781] T 0f2a4042e0f64ceebb345b9cf9764805 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:15.082271 32677 raft_consensus.cc:482] T 0f2a4042e0f64ceebb345b9cf9764805 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:15.082335 32677 raft_consensus.cc:3037] T 0f2a4042e0f64ceebb345b9cf9764805 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:15.086068 32677 raft_consensus.cc:504] T 0f2a4042e0f64ceebb345b9cf9764805 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:15.086787 32677 leader_election.cc:288] T 0f2a4042e0f64ceebb345b9cf9764805 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Requested vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:15:15.087949 26423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f2a4042e0f64ceebb345b9cf9764805" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:15:15.088194 26423 raft_consensus.cc:3037] T 0f2a4042e0f64ceebb345b9cf9764805 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:15.091713 26423 raft_consensus.cc:2454] T 0f2a4042e0f64ceebb345b9cf9764805 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:15:15.092276 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f2a4042e0f64ceebb345b9cf9764805" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:15:15.092581 26818 raft_consensus.cc:3037] T 0f2a4042e0f64ceebb345b9cf9764805 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:15.096201 26818 raft_consensus.cc:2454] T 0f2a4042e0f64ceebb345b9cf9764805 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:15:15.096781 26624 leader_election.cc:302] T 0f2a4042e0f64ceebb345b9cf9764805 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:15:15.097281 32678 raft_consensus.cc:2781] T 0f2a4042e0f64ceebb345b9cf9764805 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:15.097393 32678 raft_consensus.cc:686] T 0f2a4042e0f64ceebb345b9cf9764805 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Becoming Leader. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Running, Role: LEADER
I20240502 14:15:15.097644 32678 consensus_queue.cc:237] T 0f2a4042e0f64ceebb345b9cf9764805 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:15.099014 26281 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:15:15.102257 29155 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:15.122202 26669 tablet_service.cc:1467] Processing CreateTablet for tablet 188e62e14c8a473f8e6dafd2c877a990 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=af6c77b9d026483aa68f14e2903dbf67]), partition=
I20240502 14:15:15.122711 26669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 188e62e14c8a473f8e6dafd2c877a990. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:15.124838 29155 tablet_service.cc:1467] Processing CreateTablet for tablet 188e62e14c8a473f8e6dafd2c877a990 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=af6c77b9d026483aa68f14e2903dbf67]), partition=
I20240502 14:15:15.125228 29155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 188e62e14c8a473f8e6dafd2c877a990. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:15.129611 26797 tablet_service.cc:1467] Processing CreateTablet for tablet 188e62e14c8a473f8e6dafd2c877a990 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=af6c77b9d026483aa68f14e2903dbf67]), partition=
I20240502 14:15:15.129971 26797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 188e62e14c8a473f8e6dafd2c877a990. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:15.132261 32748 tablet_bootstrap.cc:492] T 188e62e14c8a473f8e6dafd2c877a990 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:15:15.133852 32748 tablet_bootstrap.cc:654] T 188e62e14c8a473f8e6dafd2c877a990 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:15.138387 32752 tablet_bootstrap.cc:492] T 188e62e14c8a473f8e6dafd2c877a990 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:15:15.139742 32752 tablet_bootstrap.cc:654] T 188e62e14c8a473f8e6dafd2c877a990 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:15.142947 316 tablet_bootstrap.cc:492] T 188e62e14c8a473f8e6dafd2c877a990 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:15:15.144024 26281 catalog_manager.cc:5484] T 0f2a4042e0f64ceebb345b9cf9764805 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195). New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:15.158380 316 tablet_bootstrap.cc:654] T 188e62e14c8a473f8e6dafd2c877a990 P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:15.166446 32748 tablet_bootstrap.cc:492] T 188e62e14c8a473f8e6dafd2c877a990 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:15:15.166618 32748 ts_tablet_manager.cc:1367] T 188e62e14c8a473f8e6dafd2c877a990 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.007s
I20240502 14:15:15.170465 32748 raft_consensus.cc:348] T 188e62e14c8a473f8e6dafd2c877a990 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:15.170711 32748 raft_consensus.cc:374] T 188e62e14c8a473f8e6dafd2c877a990 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:15.170768 32748 raft_consensus.cc:729] T 188e62e14c8a473f8e6dafd2c877a990 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:15:15.171813 32752 tablet_bootstrap.cc:492] T 188e62e14c8a473f8e6dafd2c877a990 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:15:15.171964 32752 ts_tablet_manager.cc:1367] T 188e62e14c8a473f8e6dafd2c877a990 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.034s user 0.006s sys 0.002s
I20240502 14:15:15.172737 32752 raft_consensus.cc:348] T 188e62e14c8a473f8e6dafd2c877a990 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:15.172983 32752 raft_consensus.cc:374] T 188e62e14c8a473f8e6dafd2c877a990 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:15.173033 32752 raft_consensus.cc:729] T 188e62e14c8a473f8e6dafd2c877a990 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:15:15.173265 32752 consensus_queue.cc:260] T 188e62e14c8a473f8e6dafd2c877a990 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:15.175395 32752 ts_tablet_manager.cc:1398] T 188e62e14c8a473f8e6dafd2c877a990 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:15:15.180994 32498 raft_consensus.cc:482] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:15.181392 32498 raft_consensus.cc:504] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:15.183125 32748 consensus_queue.cc:260] T 188e62e14c8a473f8e6dafd2c877a990 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:15.183398 32498 leader_election.cc:288] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:15:15.183542 26423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e45de4b8415446a08e9550f911367624" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:15:15.183784 26423 raft_consensus.cc:2454] T e45de4b8415446a08e9550f911367624 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 0.
I20240502 14:15:15.184010 29305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e45de4b8415446a08e9550f911367624" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:15:15.184341 29305 raft_consensus.cc:2454] T e45de4b8415446a08e9550f911367624 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 0.
I20240502 14:15:15.185017 26754 leader_election.cc:302] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1100e16b3b9495ca61d4f95e89b6a89, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:15:15.185163 316 tablet_bootstrap.cc:492] T 188e62e14c8a473f8e6dafd2c877a990 P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:15:15.185307 316 ts_tablet_manager.cc:1367] T 188e62e14c8a473f8e6dafd2c877a990 P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.002s
I20240502 14:15:15.185530 32498 raft_consensus.cc:2781] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:15.185645 32498 raft_consensus.cc:482] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:15.185758 32498 raft_consensus.cc:3037] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:15.186064 316 raft_consensus.cc:348] T 188e62e14c8a473f8e6dafd2c877a990 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:15.187312 316 raft_consensus.cc:374] T 188e62e14c8a473f8e6dafd2c877a990 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:15.187475 316 raft_consensus.cc:729] T 188e62e14c8a473f8e6dafd2c877a990 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:15:15.189532 316 consensus_queue.cc:260] T 188e62e14c8a473f8e6dafd2c877a990 P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:15.189759 32748 ts_tablet_manager.cc:1398] T 188e62e14c8a473f8e6dafd2c877a990 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20240502 14:15:15.190342 316 ts_tablet_manager.cc:1398] T 188e62e14c8a473f8e6dafd2c877a990 P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 14:15:15.202720 29421 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:15:15.189455 32498 raft_consensus.cc:504] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:15.208050 29305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e45de4b8415446a08e9550f911367624" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
I20240502 14:15:15.208467 29305 raft_consensus.cc:3037] T e45de4b8415446a08e9550f911367624 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:15.212759 29305 raft_consensus.cc:2454] T e45de4b8415446a08e9550f911367624 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
I20240502 14:15:15.216627 26754 leader_election.cc:302] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1100e16b3b9495ca61d4f95e89b6a89, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:15:15.217031 32232 raft_consensus.cc:2781] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:15.225180 32232 raft_consensus.cc:686] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Becoming Leader. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Running, Role: LEADER
I20240502 14:15:15.225567 32232 consensus_queue.cc:237] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:15.253583 32498 leader_election.cc:288] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Requested vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:15:15.254156 26423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e45de4b8415446a08e9550f911367624" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:15:15.254480 26423 raft_consensus.cc:3037] T e45de4b8415446a08e9550f911367624 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:15.262004 32607 raft_consensus.cc:482] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:15.262385 32607 raft_consensus.cc:504] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:15.263370 32607 leader_election.cc:288] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:15:15.266345 26423 raft_consensus.cc:2454] T e45de4b8415446a08e9550f911367624 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
I20240502 14:15:15.268646 29305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc3b47b4ab145ccb9efbf1b5740c7f6" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:15:15.268708 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc3b47b4ab145ccb9efbf1b5740c7f6" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:15:15.268923 26426 raft_consensus.cc:2454] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:15:15.268986 29305 raft_consensus.cc:2454] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:15:15.270962 29510 leader_election.cc:302] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:15:15.271339 32696 raft_consensus.cc:2781] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:15.271486 32696 raft_consensus.cc:482] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:15.271587 32696 raft_consensus.cc:3037] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:15.273602 26281 catalog_manager.cc:5484] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196). New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:15.274267 29554 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:15:15.276095 32696 raft_consensus.cc:504] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:15.280987 29305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc3b47b4ab145ccb9efbf1b5740c7f6" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
I20240502 14:15:15.281301 29305 raft_consensus.cc:3037] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:15.282927 32696 leader_election.cc:288] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Requested vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:15:15.283421 26423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc3b47b4ab145ccb9efbf1b5740c7f6" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:15:15.283645 26423 raft_consensus.cc:3037] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:15.285586 29305 raft_consensus.cc:2454] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:15:15.286100 29510 leader_election.cc:302] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:15:15.286527 26423 raft_consensus.cc:2454] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:15:15.291826 32607 raft_consensus.cc:2781] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:15.291975 32607 raft_consensus.cc:686] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Becoming Leader. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Running, Role: LEADER
I20240502 14:15:15.292251 32607 consensus_queue.cc:237] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:15.300719 26407 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:15.307299 26284 catalog_manager.cc:5484] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200). New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:15.334297 32232 raft_consensus.cc:482] T 188e62e14c8a473f8e6dafd2c877a990 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:15.334479 32232 raft_consensus.cc:504] T 188e62e14c8a473f8e6dafd2c877a990 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:15.335331 32232 leader_election.cc:288] T 188e62e14c8a473f8e6dafd2c877a990 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:15:15.340025 26537 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:15:15.345100 26687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "188e62e14c8a473f8e6dafd2c877a990" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:15:15.345346 26687 raft_consensus.cc:2454] T 188e62e14c8a473f8e6dafd2c877a990 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 0.
I20240502 14:15:15.345873 26755 leader_election.cc:302] T 188e62e14c8a473f8e6dafd2c877a990 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:15:15.346532 324 raft_consensus.cc:2781] T 188e62e14c8a473f8e6dafd2c877a990 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:15.346628 324 raft_consensus.cc:482] T 188e62e14c8a473f8e6dafd2c877a990 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:15.346693 324 raft_consensus.cc:3037] T 188e62e14c8a473f8e6dafd2c877a990 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:15.352572 324 raft_consensus.cc:504] T 188e62e14c8a473f8e6dafd2c877a990 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:15.353514 324 leader_election.cc:288] T 188e62e14c8a473f8e6dafd2c877a990 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Requested vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:15:15.356432 26687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "188e62e14c8a473f8e6dafd2c877a990" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:15:15.356631 26687 raft_consensus.cc:3037] T 188e62e14c8a473f8e6dafd2c877a990 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:15.361497 26687 raft_consensus.cc:2454] T 188e62e14c8a473f8e6dafd2c877a990 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
I20240502 14:15:15.363301 26755 leader_election.cc:302] T 188e62e14c8a473f8e6dafd2c877a990 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:15:15.364164 32232 raft_consensus.cc:2781] T 188e62e14c8a473f8e6dafd2c877a990 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:15.364338 32232 raft_consensus.cc:686] T 188e62e14c8a473f8e6dafd2c877a990 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Becoming Leader. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Running, Role: LEADER
I20240502 14:15:15.364658 32232 consensus_queue.cc:237] T 188e62e14c8a473f8e6dafd2c877a990 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:15.367853 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:15.384284 29176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "188e62e14c8a473f8e6dafd2c877a990" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:15:15.384579 29176 raft_consensus.cc:3037] T 188e62e14c8a473f8e6dafd2c877a990 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:15.384923 29175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "188e62e14c8a473f8e6dafd2c877a990" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:15:15.387496 29176 raft_consensus.cc:2454] T 188e62e14c8a473f8e6dafd2c877a990 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
I20240502 14:15:15.413154 29288 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:15.455139 26281 catalog_manager.cc:5484] T 188e62e14c8a473f8e6dafd2c877a990 P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196). New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:15.458833 26797 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:15.563232 32678 consensus_queue.cc:1035] T 0f2a4042e0f64ceebb345b9cf9764805 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:15.599100 26281 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:15:15.631451 29175 raft_consensus.cc:1261] T 188e62e14c8a473f8e6dafd2c877a990 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:15.632099 32232 consensus_queue.cc:1035] T 188e62e14c8a473f8e6dafd2c877a990 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:15.633523 26689 raft_consensus.cc:1261] T 188e62e14c8a473f8e6dafd2c877a990 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:15.634238 32232 consensus_queue.cc:1035] T 188e62e14c8a473f8e6dafd2c877a990 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:15.640951 26281 catalog_manager.cc:5576] TS 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715) does not have the latest schema for tablet 188e62e14c8a473f8e6dafd2c877a990 (table rebalancer_dynamic_table [id=af6c77b9d026483aa68f14e2903dbf67]). Expected version 1 got 0
I20240502 14:15:15.646631 353 tablet.cc:1722] T 188e62e14c8a473f8e6dafd2c877a990 P 1cb67158133e4648a6eceba32d4dbba3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:15.648167 353 tablet.cc:1620] T 188e62e14c8a473f8e6dafd2c877a990 P 1cb67158133e4648a6eceba32d4dbba3: MemRowSet was empty: no flush needed.
I20240502 14:15:15.650799 26284 catalog_manager.cc:5576] TS 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715) does not have the latest schema for tablet 188e62e14c8a473f8e6dafd2c877a990 (table rebalancer_dynamic_table [id=af6c77b9d026483aa68f14e2903dbf67]). Expected version 1 got 0
I20240502 14:15:15.655624 32677 consensus_queue.cc:1035] T 0f2a4042e0f64ceebb345b9cf9764805 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:15.666028 352 tablet.cc:1722] T 188e62e14c8a473f8e6dafd2c877a990 P e4cfb8fc5a474177af6848a6efa13a27: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:15.666817 352 tablet.cc:1620] T 188e62e14c8a473f8e6dafd2c877a990 P e4cfb8fc5a474177af6848a6efa13a27: MemRowSet was empty: no flush needed.
I20240502 14:15:15.686650 26284 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=af6c77b9d026483aa68f14e2903dbf67] alter complete (version 1)
I20240502 14:15:15.694334 32607 consensus_queue.cc:1035] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:15.699187 26281 catalog_manager.cc:5576] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325) does not have the latest schema for tablet 188e62e14c8a473f8e6dafd2c877a990 (table rebalancer_dynamic_table [id=af6c77b9d026483aa68f14e2903dbf67]). Expected version 1 got 0
I20240502 14:15:15.725237 32232 consensus_queue.cc:1035] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:15.729697 360 tablet.cc:1722] T 188e62e14c8a473f8e6dafd2c877a990 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:15.730321 360 tablet.cc:1620] T 188e62e14c8a473f8e6dafd2c877a990 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: MemRowSet was empty: no flush needed.
I20240502 14:15:15.752948 26281 catalog_manager.cc:5576] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325) does not have the latest schema for tablet 188e62e14c8a473f8e6dafd2c877a990 (table rebalancer_dynamic_table [id=af6c77b9d026483aa68f14e2903dbf67]). Expected version 1 got 0
I20240502 14:15:15.755134 31623 consensus_queue.cc:1035] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:15.761303 32232 consensus_queue.cc:1035] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:15.798839 26284 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:15:15.805936 26555 consensus_queue.cc:237] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:15.811196 352 tablet.cc:1722] T 188e62e14c8a473f8e6dafd2c877a990 P e4cfb8fc5a474177af6848a6efa13a27: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:15.811728 352 tablet.cc:1620] T 188e62e14c8a473f8e6dafd2c877a990 P e4cfb8fc5a474177af6848a6efa13a27: MemRowSet was empty: no flush needed.
I20240502 14:15:15.811817 360 tablet.cc:1722] T 188e62e14c8a473f8e6dafd2c877a990 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:15.812314 360 tablet.cc:1620] T 188e62e14c8a473f8e6dafd2c877a990 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: MemRowSet was empty: no flush needed.
I20240502 14:15:15.817509 26818 raft_consensus.cc:1261] T d7f02f897ce54fefb03832da3c2d95e0 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:15.819252 369 consensus_queue.cc:1035] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:15.820087 353 tablet.cc:1722] T 188e62e14c8a473f8e6dafd2c877a990 P 1cb67158133e4648a6eceba32d4dbba3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:15.820708 353 tablet.cc:1620] T 188e62e14c8a473f8e6dafd2c877a990 P 1cb67158133e4648a6eceba32d4dbba3: MemRowSet was empty: no flush needed.
I20240502 14:15:15.821805 29304 raft_consensus.cc:1261] T d7f02f897ce54fefb03832da3c2d95e0 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:15.827984 369 consensus_queue.cc:1035] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:15.828855 26284 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=af6c77b9d026483aa68f14e2903dbf67] alter complete (version 2)
I20240502 14:15:15.837692 31369 raft_consensus.cc:2932] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:15:15.838384 26819 raft_consensus.cc:2932] T d7f02f897ce54fefb03832da3c2d95e0 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:15:15.845598 26281 catalog_manager.cc:5484] T d7f02f897ce54fefb03832da3c2d95e0 P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: config changed from index 4 to 5, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:15:15.850265 26669 tablet_service.cc:1514] Processing DeleteTablet for tablet d7f02f897ce54fefb03832da3c2d95e0 with delete_type TABLET_DATA_TOMBSTONED (TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34150
I20240502 14:15:15.850055 29309 raft_consensus.cc:2932] T d7f02f897ce54fefb03832da3c2d95e0 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:15:15.852839 381 tablet_replica.cc:330] T d7f02f897ce54fefb03832da3c2d95e0 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: stopping tablet replica
I20240502 14:15:15.853113 381 raft_consensus.cc:2229] T d7f02f897ce54fefb03832da3c2d95e0 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:15.853374 381 raft_consensus.cc:2258] T d7f02f897ce54fefb03832da3c2d95e0 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:15.854171 381 ts_tablet_manager.cc:1875] T d7f02f897ce54fefb03832da3c2d95e0 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:15.860483 381 ts_tablet_manager.cc:1888] T d7f02f897ce54fefb03832da3c2d95e0 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:15:15.860729 381 log.cc:1196] T d7f02f897ce54fefb03832da3c2d95e0 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/wals/d7f02f897ce54fefb03832da3c2d95e0
I20240502 14:15:15.861539 26272 catalog_manager.cc:4830] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): tablet d7f02f897ce54fefb03832da3c2d95e0 (table rebalance_test_table_7 [id=998b08830b5f4b6fbb49d494f21739b9]) successfully deleted
I20240502 14:15:15.892349 26281 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_extra_table_50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:15.903136 29155 tablet_service.cc:1467] Processing CreateTablet for tablet 5e66bc4ba1f84b50884d6492edb1e0d2 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=183571cb1d8d40aaa35266ced8ac1c76]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:15.903645 29155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e66bc4ba1f84b50884d6492edb1e0d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:15.909191 29421 tablet_service.cc:1467] Processing CreateTablet for tablet 5e66bc4ba1f84b50884d6492edb1e0d2 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=183571cb1d8d40aaa35266ced8ac1c76]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:15.909737 29421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e66bc4ba1f84b50884d6492edb1e0d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:15.913486 26797 tablet_service.cc:1467] Processing CreateTablet for tablet 04f3eff43b544f33b020bd74e4331dea (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=183571cb1d8d40aaa35266ced8ac1c76]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:15.913939 26797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04f3eff43b544f33b020bd74e4331dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:15.914956 26537 tablet_service.cc:1467] Processing CreateTablet for tablet 5e66bc4ba1f84b50884d6492edb1e0d2 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=183571cb1d8d40aaa35266ced8ac1c76]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:15.915390 26537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e66bc4ba1f84b50884d6492edb1e0d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:15.916749 26798 tablet_service.cc:1467] Processing CreateTablet for tablet ffaded0bc6204e8c8576ea8eabdd18ab (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=183571cb1d8d40aaa35266ced8ac1c76]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:15.917106 26798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffaded0bc6204e8c8576ea8eabdd18ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:15.919973 26538 tablet_service.cc:1467] Processing CreateTablet for tablet 04f3eff43b544f33b020bd74e4331dea (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=183571cb1d8d40aaa35266ced8ac1c76]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:15.920423 26538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04f3eff43b544f33b020bd74e4331dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:15.921166 29554 tablet_service.cc:1467] Processing CreateTablet for tablet 04f3eff43b544f33b020bd74e4331dea (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=183571cb1d8d40aaa35266ced8ac1c76]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:15.921598 29554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04f3eff43b544f33b020bd74e4331dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:15.923115 29422 tablet_service.cc:1467] Processing CreateTablet for tablet ffaded0bc6204e8c8576ea8eabdd18ab (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=183571cb1d8d40aaa35266ced8ac1c76]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:15.923545 29422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffaded0bc6204e8c8576ea8eabdd18ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:15.924800 29553 tablet_service.cc:1467] Processing CreateTablet for tablet ffaded0bc6204e8c8576ea8eabdd18ab (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=183571cb1d8d40aaa35266ced8ac1c76]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:15.925222 29553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffaded0bc6204e8c8576ea8eabdd18ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:15.937893 26284 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:15:15.953380 394 tablet_bootstrap.cc:492] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:15:15.955457 394 tablet_bootstrap.cc:654] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:15.956492 393 tablet_bootstrap.cc:492] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:15:15.956558 352 tablet.cc:1722] T 188e62e14c8a473f8e6dafd2c877a990 P e4cfb8fc5a474177af6848a6efa13a27: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:15.962473 353 tablet.cc:1722] T 188e62e14c8a473f8e6dafd2c877a990 P 1cb67158133e4648a6eceba32d4dbba3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:15.965759 394 tablet_bootstrap.cc:492] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:15:15.965868 393 tablet_bootstrap.cc:654] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:15.966154 394 ts_tablet_manager.cc:1367] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20240502 14:15:15.967520 392 tablet_bootstrap.cc:492] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:15:15.967538 394 raft_consensus.cc:348] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:15.968458 394 raft_consensus.cc:374] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:15.968609 394 raft_consensus.cc:729] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:15:15.969156 394 consensus_queue.cc:260] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:15.970714 392 tablet_bootstrap.cc:654] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:15.971251 394 ts_tablet_manager.cc:1398] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:15:15.971596 394 tablet_bootstrap.cc:492] T 04f3eff43b544f33b020bd74e4331dea P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:15:15.973066 394 tablet_bootstrap.cc:654] T 04f3eff43b544f33b020bd74e4331dea P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:15.975978 391 tablet_bootstrap.cc:492] T ffaded0bc6204e8c8576ea8eabdd18ab P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:15:15.980363 360 tablet.cc:1722] T 188e62e14c8a473f8e6dafd2c877a990 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:15.984015 392 tablet_bootstrap.cc:492] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:15:15.985134 32232 raft_consensus.cc:482] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:15.985292 32232 raft_consensus.cc:504] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:15.985958 32232 leader_election.cc:288] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:15:15.986011 394 tablet_bootstrap.cc:492] T 04f3eff43b544f33b020bd74e4331dea P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:15:15.986140 394 ts_tablet_manager.cc:1367] T 04f3eff43b544f33b020bd74e4331dea P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.007s
W20240502 14:15:15.987056 26755 leader_election.cc:341] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097): Not found: Tablet not found: ffaded0bc6204e8c8576ea8eabdd18ab
I20240502 14:15:15.986975 394 raft_consensus.cc:348] T 04f3eff43b544f33b020bd74e4331dea P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:15.987164 394 raft_consensus.cc:374] T 04f3eff43b544f33b020bd74e4331dea P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:15.987324 394 raft_consensus.cc:729] T 04f3eff43b544f33b020bd74e4331dea P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:15:15.984220 392 ts_tablet_manager.cc:1367] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20240502 14:15:15.987665 394 consensus_queue.cc:260] T 04f3eff43b544f33b020bd74e4331dea P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:15.989744 394 ts_tablet_manager.cc:1398] T 04f3eff43b544f33b020bd74e4331dea P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 14:15:15.990233 392 raft_consensus.cc:348] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:15.990798 392 raft_consensus.cc:374] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:15.991595 391 tablet_bootstrap.cc:654] T ffaded0bc6204e8c8576ea8eabdd18ab P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:15.992022 392 raft_consensus.cc:729] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:15:15.993036 393 tablet_bootstrap.cc:492] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:15:15.993223 393 ts_tablet_manager.cc:1367] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.004s
I20240502 14:15:15.994277 26284 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table_new_name [id=af6c77b9d026483aa68f14e2903dbf67] alter complete (version 3)
I20240502 14:15:15.995688 393 raft_consensus.cc:348] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:15.995975 393 raft_consensus.cc:374] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:15.996119 393 raft_consensus.cc:729] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:15:15.996451 393 consensus_queue.cc:260] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:15.997258 393 ts_tablet_manager.cc:1398] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:15.991608 29439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffaded0bc6204e8c8576ea8eabdd18ab" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
W20240502 14:15:15.997987 26755 leader_election.cc:341] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:15.998126 26755 leader_election.cc:302] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4cfb8fc5a474177af6848a6efa13a27; no voters: 4092902b3e674857a0147f1b4f2cdb86, 68a3c2b17ed94e06aba66ed4178132ab
I20240502 14:15:15.998567 324 raft_consensus.cc:2726] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:15:15.997888 392 consensus_queue.cc:260] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:15.999157 392 ts_tablet_manager.cc:1398] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240502 14:15:15.999420 392 tablet_bootstrap.cc:492] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:15:16.000859 392 tablet_bootstrap.cc:654] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:16.009048 391 tablet_bootstrap.cc:492] T ffaded0bc6204e8c8576ea8eabdd18ab P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:15:16.009336 391 ts_tablet_manager.cc:1367] T ffaded0bc6204e8c8576ea8eabdd18ab P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.033s user 0.011s sys 0.000s
I20240502 14:15:16.010241 391 raft_consensus.cc:348] T ffaded0bc6204e8c8576ea8eabdd18ab P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:16.010562 391 raft_consensus.cc:374] T ffaded0bc6204e8c8576ea8eabdd18ab P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:16.010744 391 raft_consensus.cc:729] T ffaded0bc6204e8c8576ea8eabdd18ab P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:15:16.011142 391 consensus_queue.cc:260] T ffaded0bc6204e8c8576ea8eabdd18ab P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:16.012116 391 ts_tablet_manager.cc:1398] T ffaded0bc6204e8c8576ea8eabdd18ab P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:15:16.012640 391 tablet_bootstrap.cc:492] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:15:16.015367 391 tablet_bootstrap.cc:654] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:16.024874 391 tablet_bootstrap.cc:492] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:15:16.025260 391 ts_tablet_manager.cc:1367] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20240502 14:15:16.027745 391 raft_consensus.cc:348] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:16.028200 391 raft_consensus.cc:374] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:16.028324 391 raft_consensus.cc:729] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:15:16.028712 391 consensus_queue.cc:260] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:16.029868 401 tablet_bootstrap.cc:492] T 04f3eff43b544f33b020bd74e4331dea P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:15:16.031420 391 ts_tablet_manager.cc:1398] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 14:15:16.033843 401 tablet_bootstrap.cc:654] T 04f3eff43b544f33b020bd74e4331dea P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:16.048835 392 tablet_bootstrap.cc:492] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:15:16.048939 401 tablet_bootstrap.cc:492] T 04f3eff43b544f33b020bd74e4331dea P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:15:16.048995 392 ts_tablet_manager.cc:1367] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.050s user 0.016s sys 0.021s
I20240502 14:15:16.049099 401 ts_tablet_manager.cc:1367] T 04f3eff43b544f33b020bd74e4331dea P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.000s
I20240502 14:15:16.049976 392 raft_consensus.cc:348] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:16.050580 392 raft_consensus.cc:374] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:16.051064 392 raft_consensus.cc:729] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:15:16.051340 392 consensus_queue.cc:260] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:16.052243 392 ts_tablet_manager.cc:1398] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:15:16.064079 401 raft_consensus.cc:348] T 04f3eff43b544f33b020bd74e4331dea P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:16.064329 401 raft_consensus.cc:374] T 04f3eff43b544f33b020bd74e4331dea P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:16.064453 401 raft_consensus.cc:729] T 04f3eff43b544f33b020bd74e4331dea P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:15:16.064754 401 consensus_queue.cc:260] T 04f3eff43b544f33b020bd74e4331dea P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:16.071815 401 ts_tablet_manager.cc:1398] T 04f3eff43b544f33b020bd74e4331dea P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.023s user 0.000s sys 0.003s
I20240502 14:15:16.072158 401 tablet_bootstrap.cc:492] T ffaded0bc6204e8c8576ea8eabdd18ab P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:15:16.075908 32263 raft_consensus.cc:482] T ffaded0bc6204e8c8576ea8eabdd18ab P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:16.076133 32263 raft_consensus.cc:504] T ffaded0bc6204e8c8576ea8eabdd18ab P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:16.077200 32263 leader_election.cc:288] T ffaded0bc6204e8c8576ea8eabdd18ab P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:15:16.078536 26819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffaded0bc6204e8c8576ea8eabdd18ab" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:15:16.078842 26819 raft_consensus.cc:2454] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 0.
I20240502 14:15:16.079551 29379 leader_election.cc:302] T ffaded0bc6204e8c8576ea8eabdd18ab P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:15:16.080538 29574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffaded0bc6204e8c8576ea8eabdd18ab" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86" is_pre_election: true
I20240502 14:15:16.081331 32263 raft_consensus.cc:2781] T ffaded0bc6204e8c8576ea8eabdd18ab P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:16.081442 32263 raft_consensus.cc:482] T ffaded0bc6204e8c8576ea8eabdd18ab P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:16.081578 32263 raft_consensus.cc:3037] T ffaded0bc6204e8c8576ea8eabdd18ab P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
W20240502 14:15:16.084630 29379 leader_election.cc:341] T ffaded0bc6204e8c8576ea8eabdd18ab P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:16.086489 32263 raft_consensus.cc:504] T ffaded0bc6204e8c8576ea8eabdd18ab P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:16.087536 32263 leader_election.cc:288] T ffaded0bc6204e8c8576ea8eabdd18ab P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 election: Requested vote from peers 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:15:16.087579 26819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffaded0bc6204e8c8576ea8eabdd18ab" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:15:16.087767 26819 raft_consensus.cc:3037] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:16.092237 26819 raft_consensus.cc:2454] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 1.
I20240502 14:15:16.095022 401 tablet_bootstrap.cc:654] T ffaded0bc6204e8c8576ea8eabdd18ab P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:16.096069 29574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffaded0bc6204e8c8576ea8eabdd18ab" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86"
W20240502 14:15:16.096865 29379 leader_election.cc:341] T ffaded0bc6204e8c8576ea8eabdd18ab P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:16.097146 29379 leader_election.cc:302] T ffaded0bc6204e8c8576ea8eabdd18ab P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, e4cfb8fc5a474177af6848a6efa13a27; no voters: 4092902b3e674857a0147f1b4f2cdb86
I20240502 14:15:16.097887 412 raft_consensus.cc:2781] T ffaded0bc6204e8c8576ea8eabdd18ab P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:16.102072 401 tablet_bootstrap.cc:492] T ffaded0bc6204e8c8576ea8eabdd18ab P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:15:16.102381 401 ts_tablet_manager.cc:1367] T ffaded0bc6204e8c8576ea8eabdd18ab P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.003s
I20240502 14:15:16.105391 401 raft_consensus.cc:348] T ffaded0bc6204e8c8576ea8eabdd18ab P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:16.105772 401 raft_consensus.cc:374] T ffaded0bc6204e8c8576ea8eabdd18ab P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:16.105867 401 raft_consensus.cc:729] T ffaded0bc6204e8c8576ea8eabdd18ab P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:15:16.106205 401 consensus_queue.cc:260] T ffaded0bc6204e8c8576ea8eabdd18ab P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:16.106976 401 ts_tablet_manager.cc:1398] T ffaded0bc6204e8c8576ea8eabdd18ab P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20240502 14:15:16.125793 26278 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:16.126010 26278 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:16.127986 412 raft_consensus.cc:686] T ffaded0bc6204e8c8576ea8eabdd18ab P 68a3c2b17ed94e06aba66ed4178132ab [term 1 LEADER]: Becoming Leader. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Running, Role: LEADER
I20240502 14:15:16.128469 412 consensus_queue.cc:237] T ffaded0bc6204e8c8576ea8eabdd18ab P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:16.129689 26278 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet 188e62e14c8a473f8e6dafd2c877a990
I20240502 14:15:16.130919 29155 tablet_service.cc:1514] Processing DeleteTablet for tablet 188e62e14c8a473f8e6dafd2c877a990 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:16 UTC) from {username='slave'} at 127.0.0.1:55202
I20240502 14:15:16.131409 26669 tablet_service.cc:1514] Processing DeleteTablet for tablet 188e62e14c8a473f8e6dafd2c877a990 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:16 UTC) from {username='slave'} at 127.0.0.1:34150
I20240502 14:15:16.131964 381 tablet_replica.cc:330] T 188e62e14c8a473f8e6dafd2c877a990 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: stopping tablet replica
I20240502 14:15:16.132205 381 raft_consensus.cc:2229] T 188e62e14c8a473f8e6dafd2c877a990 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:16.132270 26798 tablet_service.cc:1514] Processing DeleteTablet for tablet 188e62e14c8a473f8e6dafd2c877a990 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:16 UTC) from {username='slave'} at 127.0.0.1:56010
I20240502 14:15:16.132483 381 raft_consensus.cc:2258] T 188e62e14c8a473f8e6dafd2c877a990 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:16.133474 381 ts_tablet_manager.cc:1875] T 188e62e14c8a473f8e6dafd2c877a990 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:16.139684 420 tablet_replica.cc:330] T 188e62e14c8a473f8e6dafd2c877a990 P 1cb67158133e4648a6eceba32d4dbba3: stopping tablet replica
I20240502 14:15:16.139971 420 raft_consensus.cc:2229] T 188e62e14c8a473f8e6dafd2c877a990 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:16.140337 420 raft_consensus.cc:2258] T 188e62e14c8a473f8e6dafd2c877a990 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:16.142781 26281 catalog_manager.cc:5484] T ffaded0bc6204e8c8576ea8eabdd18ab P 68a3c2b17ed94e06aba66ed4178132ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199). New cstate: current_term: 1 leader_uuid: "68a3c2b17ed94e06aba66ed4178132ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:16.145941 381 ts_tablet_manager.cc:1888] T 188e62e14c8a473f8e6dafd2c877a990 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:16.146183 381 log.cc:1196] T 188e62e14c8a473f8e6dafd2c877a990 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/wals/188e62e14c8a473f8e6dafd2c877a990
I20240502 14:15:16.146682 381 ts_tablet_manager.cc:1909] T 188e62e14c8a473f8e6dafd2c877a990 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting consensus metadata
I20240502 14:15:16.149209 26272 catalog_manager.cc:4830] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): tablet 188e62e14c8a473f8e6dafd2c877a990 (table rebalancer_dynamic_table_new_name [id=af6c77b9d026483aa68f14e2903dbf67]) successfully deleted
I20240502 14:15:16.149645 420 ts_tablet_manager.cc:1875] T 188e62e14c8a473f8e6dafd2c877a990 P 1cb67158133e4648a6eceba32d4dbba3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:16.160742 420 ts_tablet_manager.cc:1888] T 188e62e14c8a473f8e6dafd2c877a990 P 1cb67158133e4648a6eceba32d4dbba3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:16.160980 420 log.cc:1196] T 188e62e14c8a473f8e6dafd2c877a990 P 1cb67158133e4648a6eceba32d4dbba3: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal/wals/188e62e14c8a473f8e6dafd2c877a990
I20240502 14:15:16.161612 420 ts_tablet_manager.cc:1909] T 188e62e14c8a473f8e6dafd2c877a990 P 1cb67158133e4648a6eceba32d4dbba3: Deleting consensus metadata
I20240502 14:15:16.162766 26271 catalog_manager.cc:4830] TS 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): tablet 188e62e14c8a473f8e6dafd2c877a990 (table rebalancer_dynamic_table_new_name [id=af6c77b9d026483aa68f14e2903dbf67]) successfully deleted
I20240502 14:15:16.167785 421 tablet_replica.cc:330] T 188e62e14c8a473f8e6dafd2c877a990 P e4cfb8fc5a474177af6848a6efa13a27: stopping tablet replica
I20240502 14:15:16.168123 421 raft_consensus.cc:2229] T 188e62e14c8a473f8e6dafd2c877a990 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:15:16.168901 421 raft_consensus.cc:2258] T 188e62e14c8a473f8e6dafd2c877a990 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:16.170543 421 ts_tablet_manager.cc:1875] T 188e62e14c8a473f8e6dafd2c877a990 P e4cfb8fc5a474177af6848a6efa13a27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:16.186744 421 ts_tablet_manager.cc:1888] T 188e62e14c8a473f8e6dafd2c877a990 P e4cfb8fc5a474177af6848a6efa13a27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:16.187108 421 log.cc:1196] T 188e62e14c8a473f8e6dafd2c877a990 P e4cfb8fc5a474177af6848a6efa13a27: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/wals/188e62e14c8a473f8e6dafd2c877a990
I20240502 14:15:16.189121 421 ts_tablet_manager.cc:1909] T 188e62e14c8a473f8e6dafd2c877a990 P e4cfb8fc5a474177af6848a6efa13a27: Deleting consensus metadata
I20240502 14:15:16.191934 26272 catalog_manager.cc:4830] TS e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): tablet 188e62e14c8a473f8e6dafd2c877a990 (table rebalancer_dynamic_table_new_name [id=af6c77b9d026483aa68f14e2903dbf67]) successfully deleted
I20240502 14:15:16.215391 31369 raft_consensus.cc:482] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:16.215618 31369 raft_consensus.cc:504] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:16.216706 31369 leader_election.cc:288] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:15:16.217689 29176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e66bc4ba1f84b50884d6492edb1e0d2" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:15:16.218055 29176 raft_consensus.cc:2454] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 0.
I20240502 14:15:16.218683 26493 leader_election.cc:302] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:15:16.219448 369 raft_consensus.cc:2781] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:16.219558 369 raft_consensus.cc:482] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:16.219653 369 raft_consensus.cc:3037] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:16.222368 29439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e66bc4ba1f84b50884d6492edb1e0d2" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:15:16.222682 29439 raft_consensus.cc:2454] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 0.
I20240502 14:15:16.227167 369 raft_consensus.cc:504] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:16.228443 369 leader_election.cc:288] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Requested vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:15:16.229014 29176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e66bc4ba1f84b50884d6492edb1e0d2" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:15:16.229262 29176 raft_consensus.cc:3037] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:16.230086 29439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e66bc4ba1f84b50884d6492edb1e0d2" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
I20240502 14:15:16.230324 29439 raft_consensus.cc:3037] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:16.232374 29176 raft_consensus.cc:2454] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:15:16.232888 26493 leader_election.cc:302] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:15:16.233197 369 raft_consensus.cc:2781] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:16.233261 29439 raft_consensus.cc:2454] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:15:16.233336 369 raft_consensus.cc:686] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Becoming Leader. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Running, Role: LEADER
I20240502 14:15:16.233597 369 consensus_queue.cc:237] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:16.237141 26280 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:15:16.246613 31369 raft_consensus.cc:482] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:16.247593 31369 raft_consensus.cc:504] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:16.250005 31369 leader_election.cc:288] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:15:16.252359 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04f3eff43b544f33b020bd74e4331dea" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:15:16.252621 26818 raft_consensus.cc:2454] T 04f3eff43b544f33b020bd74e4331dea P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 0.
I20240502 14:15:16.254331 26494 leader_election.cc:302] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:15:16.254812 29574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04f3eff43b544f33b020bd74e4331dea" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86" is_pre_election: true
I20240502 14:15:16.255080 29574 raft_consensus.cc:2454] T 04f3eff43b544f33b020bd74e4331dea P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 0.
I20240502 14:15:16.257740 31369 raft_consensus.cc:2781] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:16.257828 31369 raft_consensus.cc:482] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:16.257884 31369 raft_consensus.cc:3037] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:16.262799 31369 raft_consensus.cc:504] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:16.264734 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04f3eff43b544f33b020bd74e4331dea" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:15:16.264940 26818 raft_consensus.cc:3037] T 04f3eff43b544f33b020bd74e4331dea P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:16.284925 26818 raft_consensus.cc:2454] T 04f3eff43b544f33b020bd74e4331dea P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:15:16.287257 31369 leader_election.cc:288] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Requested vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:15:16.290104 26278 catalog_manager.cc:5484] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a reported cstate change: term changed from 0 to 1, leader changed from <none> to ba187e38a2754d60a12de9ac4856430a (127.19.94.194). New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:16.291275 26669 tablet_service.cc:1467] Processing CreateTablet for tablet d10e96c7654c497b9dc86710002791de (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4a5c72d79f1d48c4898af2b9deb779e4]), partition=
I20240502 14:15:16.291316 29288 tablet_service.cc:1467] Processing CreateTablet for tablet d10e96c7654c497b9dc86710002791de (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4a5c72d79f1d48c4898af2b9deb779e4]), partition=
I20240502 14:15:16.291528 29155 tablet_service.cc:1467] Processing CreateTablet for tablet d10e96c7654c497b9dc86710002791de (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4a5c72d79f1d48c4898af2b9deb779e4]), partition=
I20240502 14:15:16.291656 26669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d10e96c7654c497b9dc86710002791de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:16.291671 29288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d10e96c7654c497b9dc86710002791de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:16.291899 29155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d10e96c7654c497b9dc86710002791de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:16.313288 437 tablet_bootstrap.cc:492] T d10e96c7654c497b9dc86710002791de P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:15:16.315023 437 tablet_bootstrap.cc:654] T d10e96c7654c497b9dc86710002791de P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:16.315577 436 tablet_bootstrap.cc:492] T d10e96c7654c497b9dc86710002791de P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:15:16.317646 436 tablet_bootstrap.cc:654] T d10e96c7654c497b9dc86710002791de P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:16.329927 437 tablet_bootstrap.cc:492] T d10e96c7654c497b9dc86710002791de P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:15:16.330880 437 ts_tablet_manager.cc:1367] T d10e96c7654c497b9dc86710002791de P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.010s
I20240502 14:15:16.331761 436 tablet_bootstrap.cc:492] T d10e96c7654c497b9dc86710002791de P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:15:16.331966 436 ts_tablet_manager.cc:1367] T d10e96c7654c497b9dc86710002791de P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20240502 14:15:16.332095 437 raft_consensus.cc:348] T d10e96c7654c497b9dc86710002791de P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:16.333401 437 raft_consensus.cc:374] T d10e96c7654c497b9dc86710002791de P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:16.333673 437 raft_consensus.cc:729] T d10e96c7654c497b9dc86710002791de P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:15:16.333704 436 raft_consensus.cc:348] T d10e96c7654c497b9dc86710002791de P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:16.334017 436 raft_consensus.cc:374] T d10e96c7654c497b9dc86710002791de P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:16.334121 436 raft_consensus.cc:729] T d10e96c7654c497b9dc86710002791de P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:15:16.334133 437 consensus_queue.cc:260] T d10e96c7654c497b9dc86710002791de P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:16.334463 436 consensus_queue.cc:260] T d10e96c7654c497b9dc86710002791de P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:16.335093 436 ts_tablet_manager.cc:1398] T d10e96c7654c497b9dc86710002791de P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:15:16.336521 437 ts_tablet_manager.cc:1398] T d10e96c7654c497b9dc86710002791de P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 14:15:16.372637 393 tablet_bootstrap.cc:492] T d10e96c7654c497b9dc86710002791de P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:15:16.374660 393 tablet_bootstrap.cc:654] T d10e96c7654c497b9dc86710002791de P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:16.388195 31623 raft_consensus.cc:482] T ffaded0bc6204e8c8576ea8eabdd18ab P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:16.388456 31623 raft_consensus.cc:504] T ffaded0bc6204e8c8576ea8eabdd18ab P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:16.389312 31623 leader_election.cc:288] T ffaded0bc6204e8c8576ea8eabdd18ab P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
W20240502 14:15:16.405475 29511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.199:41781, user_credentials={real_user=slave}} blocked reactor thread for 129293us
I20240502 14:15:16.406814 393 tablet_bootstrap.cc:492] T d10e96c7654c497b9dc86710002791de P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:15:16.406976 29439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffaded0bc6204e8c8576ea8eabdd18ab" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:15:16.407238 393 ts_tablet_manager.cc:1367] T d10e96c7654c497b9dc86710002791de P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.035s user 0.005s sys 0.005s
I20240502 14:15:16.407871 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffaded0bc6204e8c8576ea8eabdd18ab" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:15:16.408123 26818 raft_consensus.cc:2379] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4092902b3e674857a0147f1b4f2cdb86 in current term 1: Already voted for candidate 68a3c2b17ed94e06aba66ed4178132ab in this term.
I20240502 14:15:16.409252 393 raft_consensus.cc:348] T d10e96c7654c497b9dc86710002791de P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:16.409662 393 raft_consensus.cc:374] T d10e96c7654c497b9dc86710002791de P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:16.409937 393 raft_consensus.cc:729] T d10e96c7654c497b9dc86710002791de P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:15:16.410416 393 consensus_queue.cc:260] T d10e96c7654c497b9dc86710002791de P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:16.415004 29511 leader_election.cc:302] T ffaded0bc6204e8c8576ea8eabdd18ab P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86; no voters: 68a3c2b17ed94e06aba66ed4178132ab, e4cfb8fc5a474177af6848a6efa13a27
I20240502 14:15:16.415330 32696 raft_consensus.cc:3037] T ffaded0bc6204e8c8576ea8eabdd18ab P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:16.418072 32696 raft_consensus.cc:2726] T ffaded0bc6204e8c8576ea8eabdd18ab P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 14:15:16.420176 26494 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 137031us
I20240502 14:15:16.420511 32736 raft_consensus.cc:482] T d10e96c7654c497b9dc86710002791de P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:16.420675 32736 raft_consensus.cc:504] T d10e96c7654c497b9dc86710002791de P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:16.421694 32736 leader_election.cc:288] T d10e96c7654c497b9dc86710002791de P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:15:16.422286 29304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d10e96c7654c497b9dc86710002791de" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:15:16.422577 29304 raft_consensus.cc:2454] T d10e96c7654c497b9dc86710002791de P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 0.
I20240502 14:15:16.423266 26687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d10e96c7654c497b9dc86710002791de" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:15:16.423445 26687 raft_consensus.cc:2454] T d10e96c7654c497b9dc86710002791de P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 0.
I20240502 14:15:16.423887 29112 leader_election.cc:302] T d10e96c7654c497b9dc86710002791de P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters:
I20240502 14:15:16.424425 32736 raft_consensus.cc:2781] T d10e96c7654c497b9dc86710002791de P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:16.424538 32736 raft_consensus.cc:482] T d10e96c7654c497b9dc86710002791de P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:16.424633 32736 raft_consensus.cc:3037] T d10e96c7654c497b9dc86710002791de P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:16.426952 32736 raft_consensus.cc:504] T d10e96c7654c497b9dc86710002791de P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:16.428030 26687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d10e96c7654c497b9dc86710002791de" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:15:16.428217 26687 raft_consensus.cc:3037] T d10e96c7654c497b9dc86710002791de P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:16.428287 29304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d10e96c7654c497b9dc86710002791de" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
I20240502 14:15:16.428543 29304 raft_consensus.cc:3037] T d10e96c7654c497b9dc86710002791de P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:16.431105 32736 leader_election.cc:288] T d10e96c7654c497b9dc86710002791de P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 election: Requested vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:15:16.431244 26687 raft_consensus.cc:2454] T d10e96c7654c497b9dc86710002791de P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 1.
I20240502 14:15:16.432119 29112 leader_election.cc:302] T d10e96c7654c497b9dc86710002791de P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters:
I20240502 14:15:16.437747 29304 raft_consensus.cc:2454] T d10e96c7654c497b9dc86710002791de P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 1.
I20240502 14:15:16.445067 32736 raft_consensus.cc:2781] T d10e96c7654c497b9dc86710002791de P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:16.445364 32736 raft_consensus.cc:686] T d10e96c7654c497b9dc86710002791de P 1cb67158133e4648a6eceba32d4dbba3 [term 1 LEADER]: Becoming Leader. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Running, Role: LEADER
I20240502 14:15:16.445672 32736 consensus_queue.cc:237] T d10e96c7654c497b9dc86710002791de P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:16.453143 29574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04f3eff43b544f33b020bd74e4331dea" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86"
I20240502 14:15:16.453385 29574 raft_consensus.cc:3037] T 04f3eff43b544f33b020bd74e4331dea P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:16.456143 29574 raft_consensus.cc:2454] T 04f3eff43b544f33b020bd74e4331dea P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:15:16.465099 26280 catalog_manager.cc:5484] T d10e96c7654c497b9dc86710002791de P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197). New cstate: current_term: 1 leader_uuid: "1cb67158133e4648a6eceba32d4dbba3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:16.465483 393 ts_tablet_manager.cc:1398] T d10e96c7654c497b9dc86710002791de P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.058s user 0.003s sys 0.000s
I20240502 14:15:16.468717 26494 leader_election.cc:302] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:15:16.469623 32650 raft_consensus.cc:2781] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:16.469748 32650 raft_consensus.cc:686] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Becoming Leader. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Running, Role: LEADER
I20240502 14:15:16.470012 32650 consensus_queue.cc:237] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:16.502179 26280 catalog_manager.cc:5484] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a reported cstate change: term changed from 0 to 1, leader changed from <none> to ba187e38a2754d60a12de9ac4856430a (127.19.94.194). New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:15:16.615752 369 consensus_queue.cc:1035] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:16.621035 32650 consensus_queue.cc:1035] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:16.624379 26282 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:15:16.637802 416 consensus_queue.cc:1035] T ffaded0bc6204e8c8576ea8eabdd18ab P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:16.643483 26689 raft_consensus.cc:1261] T d10e96c7654c497b9dc86710002791de P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:16.653512 355 consensus_queue.cc:1035] T d10e96c7654c497b9dc86710002791de P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:16.655130 29304 raft_consensus.cc:1261] T d10e96c7654c497b9dc86710002791de P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:16.663663 355 consensus_queue.cc:1035] T d10e96c7654c497b9dc86710002791de P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:16.684338 412 consensus_queue.cc:1035] T ffaded0bc6204e8c8576ea8eabdd18ab P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:16.775115 26282 catalog_manager.cc:5576] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325) does not have the latest schema for tablet d10e96c7654c497b9dc86710002791de (table rebalancer_dynamic_table [id=4a5c72d79f1d48c4898af2b9deb779e4]). Expected version 1 got 0
I20240502 14:15:16.781739 29554 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:15:16.791859 26282 catalog_manager.cc:5576] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659) does not have the latest schema for tablet d10e96c7654c497b9dc86710002791de (table rebalancer_dynamic_table [id=4a5c72d79f1d48c4898af2b9deb779e4]). Expected version 1 got 0
I20240502 14:15:16.824999 490 tablet.cc:1722] T d10e96c7654c497b9dc86710002791de P 1cb67158133e4648a6eceba32d4dbba3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:16.825675 491 tablet.cc:1722] T d10e96c7654c497b9dc86710002791de P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:16.832216 491 tablet.cc:1620] T d10e96c7654c497b9dc86710002791de P c1100e16b3b9495ca61d4f95e89b6a89: MemRowSet was empty: no flush needed.
I20240502 14:15:16.832536 490 tablet.cc:1620] T d10e96c7654c497b9dc86710002791de P 1cb67158133e4648a6eceba32d4dbba3: MemRowSet was empty: no flush needed.
I20240502 14:15:16.845309 26282 catalog_manager.cc:5576] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659) does not have the latest schema for tablet d10e96c7654c497b9dc86710002791de (table rebalancer_dynamic_table [id=4a5c72d79f1d48c4898af2b9deb779e4]). Expected version 1 got 0
I20240502 14:15:16.849187 495 tablet.cc:1722] T d10e96c7654c497b9dc86710002791de P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:16.851770 495 tablet.cc:1620] T d10e96c7654c497b9dc86710002791de P 69ab78b2a1b74d2bbf2cdeafa0a4af23: MemRowSet was empty: no flush needed.
I20240502 14:15:16.853495 29155 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:15:16.857610 26282 catalog_manager.cc:5576] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325) does not have the latest schema for tablet d10e96c7654c497b9dc86710002791de (table rebalancer_dynamic_table [id=4a5c72d79f1d48c4898af2b9deb779e4]). Expected version 1 got 0
I20240502 14:15:16.869469 26281 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=4a5c72d79f1d48c4898af2b9deb779e4] alter complete (version 1)
I20240502 14:15:16.890619 26407 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:16.893510 29422 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:15:16.893623 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:16.917132 29288 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:16.949028 26538 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:15:16.958811 369 consensus_queue.cc:1035] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:16.981386 26282 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:15:16.984004 435 consensus_queue.cc:1035] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:16.995544 490 tablet.cc:1722] T d10e96c7654c497b9dc86710002791de P 1cb67158133e4648a6eceba32d4dbba3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:16.996062 490 tablet.cc:1620] T d10e96c7654c497b9dc86710002791de P 1cb67158133e4648a6eceba32d4dbba3: MemRowSet was empty: no flush needed.
I20240502 14:15:16.996845 495 tablet.cc:1722] T d10e96c7654c497b9dc86710002791de P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:16.997334 495 tablet.cc:1620] T d10e96c7654c497b9dc86710002791de P 69ab78b2a1b74d2bbf2cdeafa0a4af23: MemRowSet was empty: no flush needed.
I20240502 14:15:17.011905 26282 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=4a5c72d79f1d48c4898af2b9deb779e4] alter complete (version 2)
I20240502 14:15:17.013753 491 tablet.cc:1722] T d10e96c7654c497b9dc86710002791de P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:17.014312 491 tablet.cc:1620] T d10e96c7654c497b9dc86710002791de P c1100e16b3b9495ca61d4f95e89b6a89: MemRowSet was empty: no flush needed.
I20240502 14:15:17.020658 26798 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:15:17.033910 26281 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_extra_table_52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:17.047282 29155 tablet_service.cc:1467] Processing CreateTablet for tablet e0344791cacf4567a6c301a7b31e6451 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=65a874546db5437ea59e74602d9071de]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:17.047448 29554 tablet_service.cc:1467] Processing CreateTablet for tablet e0344791cacf4567a6c301a7b31e6451 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=65a874546db5437ea59e74602d9071de]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:17.047838 29155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0344791cacf4567a6c301a7b31e6451. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:17.047899 29554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0344791cacf4567a6c301a7b31e6451. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:17.048792 26669 tablet_service.cc:1467] Processing CreateTablet for tablet e0344791cacf4567a6c301a7b31e6451 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=65a874546db5437ea59e74602d9071de]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:17.049257 26669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0344791cacf4567a6c301a7b31e6451. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:17.054800 29288 tablet_service.cc:1467] Processing CreateTablet for tablet 63b0005e85cc4c4fab7ba8f2c728ffc2 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=65a874546db5437ea59e74602d9071de]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:17.055302 29288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63b0005e85cc4c4fab7ba8f2c728ffc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:17.055920 26538 tablet_service.cc:1467] Processing CreateTablet for tablet 842d4e7e5ee74e309fb7eba6d7b7ae8b (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=65a874546db5437ea59e74602d9071de]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:17.055941 29287 tablet_service.cc:1467] Processing CreateTablet for tablet 842d4e7e5ee74e309fb7eba6d7b7ae8b (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=65a874546db5437ea59e74602d9071de]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:17.056300 26538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 842d4e7e5ee74e309fb7eba6d7b7ae8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:17.056325 29287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 842d4e7e5ee74e309fb7eba6d7b7ae8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:17.059252 26798 tablet_service.cc:1467] Processing CreateTablet for tablet 63b0005e85cc4c4fab7ba8f2c728ffc2 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=65a874546db5437ea59e74602d9071de]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:17.059657 26798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63b0005e85cc4c4fab7ba8f2c728ffc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:17.066264 29421 tablet_service.cc:1467] Processing CreateTablet for tablet 842d4e7e5ee74e309fb7eba6d7b7ae8b (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=65a874546db5437ea59e74602d9071de]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:17.066437 29422 tablet_service.cc:1467] Processing CreateTablet for tablet 63b0005e85cc4c4fab7ba8f2c728ffc2 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=65a874546db5437ea59e74602d9071de]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:17.066756 29421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 842d4e7e5ee74e309fb7eba6d7b7ae8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:17.067096 29422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63b0005e85cc4c4fab7ba8f2c728ffc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:17.096827 507 tablet_bootstrap.cc:492] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:15:17.098888 507 tablet_bootstrap.cc:654] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:17.103349 509 tablet_bootstrap.cc:492] T e0344791cacf4567a6c301a7b31e6451 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:15:17.105583 509 tablet_bootstrap.cc:654] T e0344791cacf4567a6c301a7b31e6451 P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:17.107352 505 tablet_bootstrap.cc:492] T e0344791cacf4567a6c301a7b31e6451 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:15:17.112006 505 tablet_bootstrap.cc:654] T e0344791cacf4567a6c301a7b31e6451 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:17.114166 503 tablet_bootstrap.cc:492] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:15:17.116933 26281 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:15:17.124361 510 tablet_bootstrap.cc:492] T e0344791cacf4567a6c301a7b31e6451 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:15:17.125892 508 tablet_bootstrap.cc:492] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:15:17.129280 490 tablet.cc:1722] T d10e96c7654c497b9dc86710002791de P 1cb67158133e4648a6eceba32d4dbba3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:17.133608 495 tablet.cc:1722] T d10e96c7654c497b9dc86710002791de P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:17.137673 491 tablet.cc:1722] T d10e96c7654c497b9dc86710002791de P c1100e16b3b9495ca61d4f95e89b6a89: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:17.144248 504 tablet_bootstrap.cc:492] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:15:17.147899 26282 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table_new_name [id=4a5c72d79f1d48c4898af2b9deb779e4] alter complete (version 3)
I20240502 14:15:17.156831 507 tablet_bootstrap.cc:492] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:15:17.157227 504 tablet_bootstrap.cc:654] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:17.157872 509 tablet_bootstrap.cc:492] T e0344791cacf4567a6c301a7b31e6451 P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:15:17.157868 510 tablet_bootstrap.cc:654] T e0344791cacf4567a6c301a7b31e6451 P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:17.159960 503 tablet_bootstrap.cc:654] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:17.160205 509 ts_tablet_manager.cc:1367] T e0344791cacf4567a6c301a7b31e6451 P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.057s user 0.004s sys 0.000s
I20240502 14:15:17.160467 507 ts_tablet_manager.cc:1367] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.064s user 0.001s sys 0.003s
I20240502 14:15:17.161336 509 raft_consensus.cc:348] T e0344791cacf4567a6c301a7b31e6451 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:17.162427 507 raft_consensus.cc:348] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:17.162741 509 raft_consensus.cc:374] T e0344791cacf4567a6c301a7b31e6451 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:17.162889 509 raft_consensus.cc:729] T e0344791cacf4567a6c301a7b31e6451 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:15:17.163038 507 raft_consensus.cc:374] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:17.163439 507 raft_consensus.cc:729] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:15:17.163331 509 consensus_queue.cc:260] T e0344791cacf4567a6c301a7b31e6451 P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:17.164002 507 consensus_queue.cc:260] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:17.174871 508 tablet_bootstrap.cc:654] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:17.181455 509 ts_tablet_manager.cc:1398] T e0344791cacf4567a6c301a7b31e6451 P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20240502 14:15:17.177033 507 ts_tablet_manager.cc:1398] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240502 14:15:17.183106 505 tablet_bootstrap.cc:492] T e0344791cacf4567a6c301a7b31e6451 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:15:17.183317 505 ts_tablet_manager.cc:1367] T e0344791cacf4567a6c301a7b31e6451 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.076s user 0.000s sys 0.004s
I20240502 14:15:17.184242 505 raft_consensus.cc:348] T e0344791cacf4567a6c301a7b31e6451 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:17.184567 505 raft_consensus.cc:374] T e0344791cacf4567a6c301a7b31e6451 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:17.185747 505 raft_consensus.cc:729] T e0344791cacf4567a6c301a7b31e6451 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:15:17.186089 505 consensus_queue.cc:260] T e0344791cacf4567a6c301a7b31e6451 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:17.188067 505 ts_tablet_manager.cc:1398] T e0344791cacf4567a6c301a7b31e6451 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 14:15:17.193161 507 tablet_bootstrap.cc:492] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:15:17.195899 507 tablet_bootstrap.cc:654] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:17.197942 503 tablet_bootstrap.cc:492] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:15:17.198087 503 ts_tablet_manager.cc:1367] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.084s user 0.003s sys 0.000s
I20240502 14:15:17.198786 503 raft_consensus.cc:348] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:17.198956 503 raft_consensus.cc:374] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:17.199292 503 raft_consensus.cc:729] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:15:17.199607 503 consensus_queue.cc:260] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:17.199965 507 tablet_bootstrap.cc:492] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:15:17.200167 507 ts_tablet_manager.cc:1367] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20240502 14:15:17.201035 507 raft_consensus.cc:348] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:17.201332 507 raft_consensus.cc:374] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:17.201479 507 raft_consensus.cc:729] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:15:17.201833 507 consensus_queue.cc:260] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:17.203809 507 ts_tablet_manager.cc:1398] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 14:15:17.209926 504 tablet_bootstrap.cc:492] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:15:17.210109 504 ts_tablet_manager.cc:1367] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.066s user 0.007s sys 0.007s
I20240502 14:15:17.210484 510 tablet_bootstrap.cc:492] T e0344791cacf4567a6c301a7b31e6451 P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:15:17.210666 503 ts_tablet_manager.cc:1398] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240502 14:15:17.211055 504 raft_consensus.cc:348] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:17.211315 504 raft_consensus.cc:374] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:17.211346 508 tablet_bootstrap.cc:492] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:15:17.211477 504 raft_consensus.cc:729] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:15:17.211660 510 ts_tablet_manager.cc:1367] T e0344791cacf4567a6c301a7b31e6451 P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.087s user 0.004s sys 0.000s
I20240502 14:15:17.212522 508 ts_tablet_manager.cc:1367] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.087s user 0.001s sys 0.003s
I20240502 14:15:17.214088 504 consensus_queue.cc:260] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:17.214895 508 raft_consensus.cc:348] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:17.215245 504 ts_tablet_manager.cc:1398] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 14:15:17.215268 510 raft_consensus.cc:348] T e0344791cacf4567a6c301a7b31e6451 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:17.215543 510 raft_consensus.cc:374] T e0344791cacf4567a6c301a7b31e6451 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:17.215655 504 tablet_bootstrap.cc:492] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:15:17.215754 510 raft_consensus.cc:729] T e0344791cacf4567a6c301a7b31e6451 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:15:17.215889 508 raft_consensus.cc:374] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:17.215997 508 raft_consensus.cc:729] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:15:17.216441 510 consensus_queue.cc:260] T e0344791cacf4567a6c301a7b31e6451 P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:17.216953 508 consensus_queue.cc:260] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:17.218923 504 tablet_bootstrap.cc:654] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:17.222807 508 ts_tablet_manager.cc:1398] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240502 14:15:17.223057 510 ts_tablet_manager.cc:1398] T e0344791cacf4567a6c301a7b31e6451 P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 14:15:17.254074 504 tablet_bootstrap.cc:492] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:15:17.254364 504 ts_tablet_manager.cc:1367] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.039s user 0.000s sys 0.005s
I20240502 14:15:17.255147 504 raft_consensus.cc:348] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:17.255456 504 raft_consensus.cc:374] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:17.255630 504 raft_consensus.cc:729] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:15:17.255931 504 consensus_queue.cc:260] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:17.256652 504 ts_tablet_manager.cc:1398] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:15:17.256776 26281 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:17.256994 26281 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:17.263955 26281 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: Sending DeleteTablet for 3 replicas of tablet d10e96c7654c497b9dc86710002791de
I20240502 14:15:17.264915 26669 tablet_service.cc:1514] Processing DeleteTablet for tablet d10e96c7654c497b9dc86710002791de with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:17 UTC) from {username='slave'} at 127.0.0.1:34150
I20240502 14:15:17.265395 29287 tablet_service.cc:1514] Processing DeleteTablet for tablet d10e96c7654c497b9dc86710002791de with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:17 UTC) from {username='slave'} at 127.0.0.1:59160
I20240502 14:15:17.265786 29155 tablet_service.cc:1514] Processing DeleteTablet for tablet d10e96c7654c497b9dc86710002791de with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:17 UTC) from {username='slave'} at 127.0.0.1:55202
I20240502 14:15:17.281911 523 tablet_replica.cc:330] T d10e96c7654c497b9dc86710002791de P 1cb67158133e4648a6eceba32d4dbba3: stopping tablet replica
I20240502 14:15:17.282754 522 tablet_replica.cc:330] T d10e96c7654c497b9dc86710002791de P c1100e16b3b9495ca61d4f95e89b6a89: stopping tablet replica
I20240502 14:15:17.284104 522 raft_consensus.cc:2229] T d10e96c7654c497b9dc86710002791de P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:17.284525 522 raft_consensus.cc:2258] T d10e96c7654c497b9dc86710002791de P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:17.284549 523 raft_consensus.cc:2229] T d10e96c7654c497b9dc86710002791de P 1cb67158133e4648a6eceba32d4dbba3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:15:17.285032 523 raft_consensus.cc:2258] T d10e96c7654c497b9dc86710002791de P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:17.285665 522 ts_tablet_manager.cc:1875] T d10e96c7654c497b9dc86710002791de P c1100e16b3b9495ca61d4f95e89b6a89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:17.286445 521 tablet_replica.cc:330] T d10e96c7654c497b9dc86710002791de P 69ab78b2a1b74d2bbf2cdeafa0a4af23: stopping tablet replica
I20240502 14:15:17.286896 521 raft_consensus.cc:2229] T d10e96c7654c497b9dc86710002791de P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:17.287210 521 raft_consensus.cc:2258] T d10e96c7654c497b9dc86710002791de P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:17.287690 434 raft_consensus.cc:482] T e0344791cacf4567a6c301a7b31e6451 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:17.287860 434 raft_consensus.cc:504] T e0344791cacf4567a6c301a7b31e6451 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:17.287974 523 ts_tablet_manager.cc:1875] T d10e96c7654c497b9dc86710002791de P 1cb67158133e4648a6eceba32d4dbba3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:17.288554 521 ts_tablet_manager.cc:1875] T d10e96c7654c497b9dc86710002791de P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:17.288622 434 leader_election.cc:288] T e0344791cacf4567a6c301a7b31e6451 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:15:17.289594 29175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0344791cacf4567a6c301a7b31e6451" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:15:17.289885 29175 raft_consensus.cc:2454] T e0344791cacf4567a6c301a7b31e6451 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:15:17.290580 29510 leader_election.cc:302] T e0344791cacf4567a6c301a7b31e6451 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 4092902b3e674857a0147f1b4f2cdb86; no voters:
I20240502 14:15:17.291190 32696 raft_consensus.cc:2781] T e0344791cacf4567a6c301a7b31e6451 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:17.291283 32696 raft_consensus.cc:482] T e0344791cacf4567a6c301a7b31e6451 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:17.291410 32696 raft_consensus.cc:3037] T e0344791cacf4567a6c301a7b31e6451 P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:17.294201 32696 raft_consensus.cc:504] T e0344791cacf4567a6c301a7b31e6451 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:17.294477 523 ts_tablet_manager.cc:1888] T d10e96c7654c497b9dc86710002791de P 1cb67158133e4648a6eceba32d4dbba3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:17.294580 523 log.cc:1196] T d10e96c7654c497b9dc86710002791de P 1cb67158133e4648a6eceba32d4dbba3: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal/wals/d10e96c7654c497b9dc86710002791de
I20240502 14:15:17.294864 32696 leader_election.cc:288] T e0344791cacf4567a6c301a7b31e6451 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Requested vote from peers 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:15:17.295001 523 ts_tablet_manager.cc:1909] T d10e96c7654c497b9dc86710002791de P 1cb67158133e4648a6eceba32d4dbba3: Deleting consensus metadata
I20240502 14:15:17.289296 26689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0344791cacf4567a6c301a7b31e6451" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:15:17.296541 26689 raft_consensus.cc:2454] T e0344791cacf4567a6c301a7b31e6451 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:15:17.296205 26687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0344791cacf4567a6c301a7b31e6451" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:15:17.296846 26687 raft_consensus.cc:3037] T e0344791cacf4567a6c301a7b31e6451 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:17.297124 29176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0344791cacf4567a6c301a7b31e6451" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:15:17.297303 29176 raft_consensus.cc:3037] T e0344791cacf4567a6c301a7b31e6451 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:17.299677 26687 raft_consensus.cc:2454] T e0344791cacf4567a6c301a7b31e6451 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:15:17.299969 29176 raft_consensus.cc:2454] T e0344791cacf4567a6c301a7b31e6451 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:15:17.300451 26271 catalog_manager.cc:4830] TS 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): tablet d10e96c7654c497b9dc86710002791de (table rebalancer_dynamic_table_new_name [id=4a5c72d79f1d48c4898af2b9deb779e4]) successfully deleted
I20240502 14:15:17.300699 522 ts_tablet_manager.cc:1888] T d10e96c7654c497b9dc86710002791de P c1100e16b3b9495ca61d4f95e89b6a89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:17.300840 522 log.cc:1196] T d10e96c7654c497b9dc86710002791de P c1100e16b3b9495ca61d4f95e89b6a89: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal/wals/d10e96c7654c497b9dc86710002791de
I20240502 14:15:17.301331 522 ts_tablet_manager.cc:1909] T d10e96c7654c497b9dc86710002791de P c1100e16b3b9495ca61d4f95e89b6a89: Deleting consensus metadata
I20240502 14:15:17.301456 521 ts_tablet_manager.cc:1888] T d10e96c7654c497b9dc86710002791de P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:17.301635 29511 leader_election.cc:302] T e0344791cacf4567a6c301a7b31e6451 P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters:
I20240502 14:15:17.302496 26271 catalog_manager.cc:4830] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): tablet d10e96c7654c497b9dc86710002791de (table rebalancer_dynamic_table_new_name [id=4a5c72d79f1d48c4898af2b9deb779e4]) successfully deleted
I20240502 14:15:17.302750 525 raft_consensus.cc:2781] T e0344791cacf4567a6c301a7b31e6451 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:17.302903 525 raft_consensus.cc:686] T e0344791cacf4567a6c301a7b31e6451 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Becoming Leader. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Running, Role: LEADER
I20240502 14:15:17.303191 525 consensus_queue.cc:237] T e0344791cacf4567a6c301a7b31e6451 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:17.308503 521 log.cc:1196] T d10e96c7654c497b9dc86710002791de P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/wals/d10e96c7654c497b9dc86710002791de
I20240502 14:15:17.309087 521 ts_tablet_manager.cc:1909] T d10e96c7654c497b9dc86710002791de P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting consensus metadata
I20240502 14:15:17.310609 26272 catalog_manager.cc:4830] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): tablet d10e96c7654c497b9dc86710002791de (table rebalancer_dynamic_table_new_name [id=4a5c72d79f1d48c4898af2b9deb779e4]) successfully deleted
I20240502 14:15:17.316021 26279 catalog_manager.cc:5484] T e0344791cacf4567a6c301a7b31e6451 P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200). New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:17.321288 411 raft_consensus.cc:482] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:17.321516 411 raft_consensus.cc:504] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:17.322540 411 leader_election.cc:288] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:15:17.325763 29442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "842d4e7e5ee74e309fb7eba6d7b7ae8b" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:15:17.326128 29442 raft_consensus.cc:2454] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 0.
I20240502 14:15:17.326613 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "842d4e7e5ee74e309fb7eba6d7b7ae8b" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:15:17.326856 26558 raft_consensus.cc:2454] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 0.
I20240502 14:15:17.328699 29244 leader_election.cc:302] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:15:17.330894 416 raft_consensus.cc:482] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:17.331061 416 raft_consensus.cc:504] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:17.331771 416 leader_election.cc:288] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:15:17.332664 26819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b0005e85cc4c4fab7ba8f2c728ffc2" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:15:17.332895 26819 raft_consensus.cc:2454] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 0.
I20240502 14:15:17.333464 29379 leader_election.cc:302] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:15:17.333894 412 raft_consensus.cc:2781] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:17.333990 412 raft_consensus.cc:482] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:17.334120 412 raft_consensus.cc:3037] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:17.334681 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b0005e85cc4c4fab7ba8f2c728ffc2" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" is_pre_election: true
I20240502 14:15:17.334908 29309 raft_consensus.cc:2454] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 0.
I20240502 14:15:17.336740 411 raft_consensus.cc:2781] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:17.336866 411 raft_consensus.cc:482] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:17.336964 411 raft_consensus.cc:3037] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:17.338289 412 raft_consensus.cc:504] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:17.339362 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b0005e85cc4c4fab7ba8f2c728ffc2" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
I20240502 14:15:17.339569 29309 raft_consensus.cc:3037] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:17.340984 411 raft_consensus.cc:504] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:17.345669 29442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "842d4e7e5ee74e309fb7eba6d7b7ae8b" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
I20240502 14:15:17.345899 29442 raft_consensus.cc:3037] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:17.348958 29442 raft_consensus.cc:2454] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 1.
I20240502 14:15:17.349782 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "842d4e7e5ee74e309fb7eba6d7b7ae8b" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:15:17.349817 26819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b0005e85cc4c4fab7ba8f2c728ffc2" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:15:17.349997 26819 raft_consensus.cc:3037] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:17.350005 26558 raft_consensus.cc:3037] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:17.358971 26558 raft_consensus.cc:2454] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 1.
I20240502 14:15:17.359298 411 leader_election.cc:288] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 election: Requested vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:15:17.359757 29244 leader_election.cc:302] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:15:17.360881 411 raft_consensus.cc:2781] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:17.361084 411 raft_consensus.cc:686] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 LEADER]: Becoming Leader. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Running, Role: LEADER
I20240502 14:15:17.361423 411 consensus_queue.cc:237] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:17.361802 26819 raft_consensus.cc:2454] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 1.
I20240502 14:15:17.362018 412 leader_election.cc:288] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 election: Requested vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:15:17.362504 29379 leader_election.cc:302] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:15:17.362937 416 raft_consensus.cc:2781] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:17.363962 416 raft_consensus.cc:686] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 LEADER]: Becoming Leader. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Running, Role: LEADER
I20240502 14:15:17.364307 416 consensus_queue.cc:237] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:17.365672 29309 raft_consensus.cc:2454] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 1.
I20240502 14:15:17.367116 26279 catalog_manager.cc:5484] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P c1100e16b3b9495ca61d4f95e89b6a89 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198). New cstate: current_term: 1 leader_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:15:17.368844 26281 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:15:17.372299 26282 catalog_manager.cc:5484] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P 68a3c2b17ed94e06aba66ed4178132ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199). New cstate: current_term: 1 leader_uuid: "68a3c2b17ed94e06aba66ed4178132ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:15:17.382647 29421 tablet_service.cc:1467] Processing CreateTablet for tablet c0d152eda1384b6792246b2f10d667da (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f9780b3608294a1380a60ae3416d3829]), partition=
I20240502 14:15:17.382711 29554 tablet_service.cc:1467] Processing CreateTablet for tablet c0d152eda1384b6792246b2f10d667da (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f9780b3608294a1380a60ae3416d3829]), partition=
I20240502 14:15:17.383021 29421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0d152eda1384b6792246b2f10d667da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:17.383073 29554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0d152eda1384b6792246b2f10d667da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:17.387367 26538 tablet_service.cc:1467] Processing CreateTablet for tablet c0d152eda1384b6792246b2f10d667da (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f9780b3608294a1380a60ae3416d3829]), partition=
I20240502 14:15:17.387737 26538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0d152eda1384b6792246b2f10d667da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:17.391219 510 tablet_bootstrap.cc:492] T c0d152eda1384b6792246b2f10d667da P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:15:17.392655 510 tablet_bootstrap.cc:654] T c0d152eda1384b6792246b2f10d667da P 4092902b3e674857a0147f1b4f2cdb86: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:17.394692 510 tablet_bootstrap.cc:492] T c0d152eda1384b6792246b2f10d667da P 4092902b3e674857a0147f1b4f2cdb86: No bootstrap required, opened a new log
I20240502 14:15:17.394930 510 ts_tablet_manager.cc:1367] T c0d152eda1384b6792246b2f10d667da P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:15:17.395659 507 tablet_bootstrap.cc:492] T c0d152eda1384b6792246b2f10d667da P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:15:17.396677 510 raft_consensus.cc:348] T c0d152eda1384b6792246b2f10d667da P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:17.396921 510 raft_consensus.cc:374] T c0d152eda1384b6792246b2f10d667da P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:17.397465 510 raft_consensus.cc:729] T c0d152eda1384b6792246b2f10d667da P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:15:17.397749 510 consensus_queue.cc:260] T c0d152eda1384b6792246b2f10d667da P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:17.397940 507 tablet_bootstrap.cc:654] T c0d152eda1384b6792246b2f10d667da P 68a3c2b17ed94e06aba66ed4178132ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:17.401644 510 ts_tablet_manager.cc:1398] T c0d152eda1384b6792246b2f10d667da P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:15:17.406728 507 tablet_bootstrap.cc:492] T c0d152eda1384b6792246b2f10d667da P 68a3c2b17ed94e06aba66ed4178132ab: No bootstrap required, opened a new log
I20240502 14:15:17.406893 507 ts_tablet_manager.cc:1367] T c0d152eda1384b6792246b2f10d667da P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.003s
I20240502 14:15:17.407743 507 raft_consensus.cc:348] T c0d152eda1384b6792246b2f10d667da P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:17.407975 507 raft_consensus.cc:374] T c0d152eda1384b6792246b2f10d667da P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:17.408049 503 tablet_bootstrap.cc:492] T c0d152eda1384b6792246b2f10d667da P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:15:17.408087 507 raft_consensus.cc:729] T c0d152eda1384b6792246b2f10d667da P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:15:17.408304 507 consensus_queue.cc:260] T c0d152eda1384b6792246b2f10d667da P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:17.409154 507 ts_tablet_manager.cc:1398] T c0d152eda1384b6792246b2f10d667da P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:17.409395 503 tablet_bootstrap.cc:654] T c0d152eda1384b6792246b2f10d667da P ba187e38a2754d60a12de9ac4856430a: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:17.416900 503 tablet_bootstrap.cc:492] T c0d152eda1384b6792246b2f10d667da P ba187e38a2754d60a12de9ac4856430a: No bootstrap required, opened a new log
I20240502 14:15:17.417064 503 ts_tablet_manager.cc:1367] T c0d152eda1384b6792246b2f10d667da P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20240502 14:15:17.417810 503 raft_consensus.cc:348] T c0d152eda1384b6792246b2f10d667da P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:17.418036 503 raft_consensus.cc:374] T c0d152eda1384b6792246b2f10d667da P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:17.418118 503 raft_consensus.cc:729] T c0d152eda1384b6792246b2f10d667da P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:15:17.418373 503 consensus_queue.cc:260] T c0d152eda1384b6792246b2f10d667da P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:17.419027 503 ts_tablet_manager.cc:1398] T c0d152eda1384b6792246b2f10d667da P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:17.452376 26558 consensus_queue.cc:237] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:17.454443 29176 raft_consensus.cc:1261] T 310aa6ad3604487589dbe96afa25bb05 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:17.454982 26687 raft_consensus.cc:1261] T 310aa6ad3604487589dbe96afa25bb05 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:17.454968 435 consensus_queue.cc:1035] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:17.455505 435 consensus_queue.cc:1035] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:17.458139 435 raft_consensus.cc:2932] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:17.461395 29176 raft_consensus.cc:2932] T 310aa6ad3604487589dbe96afa25bb05 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:17.463604 26687 raft_consensus.cc:2932] T 310aa6ad3604487589dbe96afa25bb05 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:17.467772 26281 catalog_manager.cc:5484] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a reported cstate change: config changed from index 4 to 5, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:17.475731 26798 tablet_service.cc:1514] Processing DeleteTablet for tablet 310aa6ad3604487589dbe96afa25bb05 with delete_type TABLET_DATA_TOMBSTONED (TS e4cfb8fc5a474177af6848a6efa13a27 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:56010
I20240502 14:15:17.478082 552 tablet_replica.cc:330] T 310aa6ad3604487589dbe96afa25bb05 P e4cfb8fc5a474177af6848a6efa13a27: stopping tablet replica
I20240502 14:15:17.478355 552 raft_consensus.cc:2229] T 310aa6ad3604487589dbe96afa25bb05 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:17.478693 552 raft_consensus.cc:2258] T 310aa6ad3604487589dbe96afa25bb05 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:17.479589 552 ts_tablet_manager.cc:1875] T 310aa6ad3604487589dbe96afa25bb05 P e4cfb8fc5a474177af6848a6efa13a27: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:17.493120 552 ts_tablet_manager.cc:1888] T 310aa6ad3604487589dbe96afa25bb05 P e4cfb8fc5a474177af6848a6efa13a27: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:15:17.493310 552 log.cc:1196] T 310aa6ad3604487589dbe96afa25bb05 P e4cfb8fc5a474177af6848a6efa13a27: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/wals/310aa6ad3604487589dbe96afa25bb05
I20240502 14:15:17.494827 26272 catalog_manager.cc:4830] TS e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): tablet 310aa6ad3604487589dbe96afa25bb05 (table rebalance_test_table_9 [id=3c3eeaa25da94395a642aa569b652348]) successfully deleted
I20240502 14:15:17.564579 434 raft_consensus.cc:482] T c0d152eda1384b6792246b2f10d667da P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:17.564793 434 raft_consensus.cc:504] T c0d152eda1384b6792246b2f10d667da P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:17.565583 434 leader_election.cc:288] T c0d152eda1384b6792246b2f10d667da P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:15:17.567412 29442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0d152eda1384b6792246b2f10d667da" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:15:17.567503 26555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0d152eda1384b6792246b2f10d667da" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:15:17.567668 29442 raft_consensus.cc:2454] T c0d152eda1384b6792246b2f10d667da P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:15:17.567785 26555 raft_consensus.cc:2454] T c0d152eda1384b6792246b2f10d667da P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 0.
I20240502 14:15:17.568176 29511 leader_election.cc:302] T c0d152eda1384b6792246b2f10d667da P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, 68a3c2b17ed94e06aba66ed4178132ab; no voters:
I20240502 14:15:17.568761 525 raft_consensus.cc:2781] T c0d152eda1384b6792246b2f10d667da P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:17.568898 525 raft_consensus.cc:482] T c0d152eda1384b6792246b2f10d667da P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:17.569022 525 raft_consensus.cc:3037] T c0d152eda1384b6792246b2f10d667da P 4092902b3e674857a0147f1b4f2cdb86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:17.571918 525 raft_consensus.cc:504] T c0d152eda1384b6792246b2f10d667da P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:17.573401 29442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0d152eda1384b6792246b2f10d667da" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
I20240502 14:15:17.573653 29442 raft_consensus.cc:3037] T c0d152eda1384b6792246b2f10d667da P 68a3c2b17ed94e06aba66ed4178132ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:17.579752 29442 raft_consensus.cc:2454] T c0d152eda1384b6792246b2f10d667da P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:15:17.580200 29511 leader_election.cc:302] T c0d152eda1384b6792246b2f10d667da P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, 68a3c2b17ed94e06aba66ed4178132ab; no voters:
I20240502 14:15:17.580572 434 raft_consensus.cc:2781] T c0d152eda1384b6792246b2f10d667da P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:17.580726 434 raft_consensus.cc:686] T c0d152eda1384b6792246b2f10d667da P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Becoming Leader. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Running, Role: LEADER
I20240502 14:15:17.581043 434 consensus_queue.cc:237] T c0d152eda1384b6792246b2f10d667da P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:17.582995 525 leader_election.cc:288] T c0d152eda1384b6792246b2f10d667da P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 1 election: Requested vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:15:17.583534 26555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0d152eda1384b6792246b2f10d667da" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:15:17.583725 26555 raft_consensus.cc:3037] T c0d152eda1384b6792246b2f10d667da P ba187e38a2754d60a12de9ac4856430a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:17.586587 26555 raft_consensus.cc:2454] T c0d152eda1384b6792246b2f10d667da P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:15:17.593443 26282 catalog_manager.cc:5484] T c0d152eda1384b6792246b2f10d667da P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200). New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:17.701449 26280 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:15:17.728689 29443 raft_consensus.cc:1261] T c0d152eda1384b6792246b2f10d667da P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:17.728749 26555 raft_consensus.cc:1261] T c0d152eda1384b6792246b2f10d667da P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:17.729485 434 consensus_queue.cc:1035] T c0d152eda1384b6792246b2f10d667da P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:17.730122 434 consensus_queue.cc:1035] T c0d152eda1384b6792246b2f10d667da P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:17.758467 411 consensus_queue.cc:1035] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:17.782882 434 consensus_queue.cc:1035] T e0344791cacf4567a6c301a7b31e6451 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:17.786199 26282 catalog_manager.cc:5576] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025) does not have the latest schema for tablet c0d152eda1384b6792246b2f10d667da (table rebalancer_dynamic_table [id=f9780b3608294a1380a60ae3416d3829]). Expected version 1 got 0
I20240502 14:15:17.790180 26282 catalog_manager.cc:5576] TS 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781) does not have the latest schema for tablet c0d152eda1384b6792246b2f10d667da (table rebalancer_dynamic_table [id=f9780b3608294a1380a60ae3416d3829]). Expected version 1 got 0
I20240502 14:15:17.790508 603 tablet.cc:1722] T c0d152eda1384b6792246b2f10d667da P ba187e38a2754d60a12de9ac4856430a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:17.791692 603 tablet.cc:1620] T c0d152eda1384b6792246b2f10d667da P ba187e38a2754d60a12de9ac4856430a: MemRowSet was empty: no flush needed.
I20240502 14:15:17.804668 26282 catalog_manager.cc:5576] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025) does not have the latest schema for tablet c0d152eda1384b6792246b2f10d667da (table rebalancer_dynamic_table [id=f9780b3608294a1380a60ae3416d3829]). Expected version 1 got 0
I20240502 14:15:17.815387 602 tablet.cc:1722] T c0d152eda1384b6792246b2f10d667da P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:17.831176 26282 catalog_manager.cc:5576] TS 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781) does not have the latest schema for tablet c0d152eda1384b6792246b2f10d667da (table rebalancer_dynamic_table [id=f9780b3608294a1380a60ae3416d3829]). Expected version 1 got 0
I20240502 14:15:17.803329 604 tablet.cc:1722] T c0d152eda1384b6792246b2f10d667da P 68a3c2b17ed94e06aba66ed4178132ab: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:17.847715 416 consensus_queue.cc:1035] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:17.857391 32439 consensus_queue.cc:1035] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:17.864379 525 consensus_queue.cc:1035] T e0344791cacf4567a6c301a7b31e6451 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:17.873916 602 tablet.cc:1620] T c0d152eda1384b6792246b2f10d667da P 4092902b3e674857a0147f1b4f2cdb86: MemRowSet was empty: no flush needed.
I20240502 14:15:17.879897 412 consensus_queue.cc:1035] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:17.884876 604 tablet.cc:1620] T c0d152eda1384b6792246b2f10d667da P 68a3c2b17ed94e06aba66ed4178132ab: MemRowSet was empty: no flush needed.
I20240502 14:15:17.890493 26284 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42842:
name: "rebalancer_extra_table_54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:17.907178 26798 tablet_service.cc:1467] Processing CreateTablet for tablet d16a118ea4c04a58890c7dd3ff9ec7b4 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=36399b8bbc064edb98ef3de360e0e9bf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:17.907670 26798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d16a118ea4c04a58890c7dd3ff9ec7b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:17.910795 26407 tablet_service.cc:1467] Processing CreateTablet for tablet d16a118ea4c04a58890c7dd3ff9ec7b4 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=36399b8bbc064edb98ef3de360e0e9bf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:17.911229 26407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d16a118ea4c04a58890c7dd3ff9ec7b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:17.922371 26669 tablet_service.cc:1467] Processing CreateTablet for tablet d16a118ea4c04a58890c7dd3ff9ec7b4 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=36399b8bbc064edb98ef3de360e0e9bf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:17.922845 26669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d16a118ea4c04a58890c7dd3ff9ec7b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:17.928561 26797 tablet_service.cc:1467] Processing CreateTablet for tablet a42e68ab03f84d4b92893006fcf6f229 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=36399b8bbc064edb98ef3de360e0e9bf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:17.928999 26797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a42e68ab03f84d4b92893006fcf6f229. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:17.928949 29155 tablet_service.cc:1467] Processing CreateTablet for tablet a42e68ab03f84d4b92893006fcf6f229 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=36399b8bbc064edb98ef3de360e0e9bf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:17.929350 29155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a42e68ab03f84d4b92893006fcf6f229. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:17.930233 26406 tablet_service.cc:1467] Processing CreateTablet for tablet a42e68ab03f84d4b92893006fcf6f229 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=36399b8bbc064edb98ef3de360e0e9bf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:17.930682 26406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a42e68ab03f84d4b92893006fcf6f229. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:17.936046 29287 tablet_service.cc:1467] Processing CreateTablet for tablet 46aaa6186d254ff7bd493bd85479015d (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=36399b8bbc064edb98ef3de360e0e9bf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:17.936473 29287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46aaa6186d254ff7bd493bd85479015d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:17.938374 26282 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=f9780b3608294a1380a60ae3416d3829] alter complete (version 1)
I20240502 14:15:17.940230 26668 tablet_service.cc:1467] Processing CreateTablet for tablet 46aaa6186d254ff7bd493bd85479015d (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=36399b8bbc064edb98ef3de360e0e9bf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:17.940719 26668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46aaa6186d254ff7bd493bd85479015d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:17.940847 29154 tablet_service.cc:1467] Processing CreateTablet for tablet 46aaa6186d254ff7bd493bd85479015d (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=36399b8bbc064edb98ef3de360e0e9bf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:17.941229 29154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46aaa6186d254ff7bd493bd85479015d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:17.959406 621 tablet_bootstrap.cc:492] T a42e68ab03f84d4b92893006fcf6f229 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:15:17.961800 621 tablet_bootstrap.cc:654] T a42e68ab03f84d4b92893006fcf6f229 P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:17.965991 623 tablet_bootstrap.cc:492] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:15:17.968302 623 tablet_bootstrap.cc:654] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:17.975523 621 tablet_bootstrap.cc:492] T a42e68ab03f84d4b92893006fcf6f229 P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:15:17.975927 621 ts_tablet_manager.cc:1367] T a42e68ab03f84d4b92893006fcf6f229 P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.004s
I20240502 14:15:17.978268 621 raft_consensus.cc:348] T a42e68ab03f84d4b92893006fcf6f229 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:17.980173 621 raft_consensus.cc:374] T a42e68ab03f84d4b92893006fcf6f229 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:17.980808 621 raft_consensus.cc:729] T a42e68ab03f84d4b92893006fcf6f229 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:15:17.981772 621 consensus_queue.cc:260] T a42e68ab03f84d4b92893006fcf6f229 P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:17.983935 621 ts_tablet_manager.cc:1398] T a42e68ab03f84d4b92893006fcf6f229 P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240502 14:15:17.985615 621 tablet_bootstrap.cc:492] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:15:17.986671 623 tablet_bootstrap.cc:492] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:15:17.988507 621 tablet_bootstrap.cc:654] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:17.989184 623 ts_tablet_manager.cc:1367] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.000s
I20240502 14:15:17.990206 623 raft_consensus.cc:348] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:17.991684 623 raft_consensus.cc:374] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:17.992007 623 raft_consensus.cc:729] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:15:17.993585 623 consensus_queue.cc:260] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:17.996248 623 ts_tablet_manager.cc:1398] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:15:17.996968 623 tablet_bootstrap.cc:492] T d16a118ea4c04a58890c7dd3ff9ec7b4 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:15:17.999003 623 tablet_bootstrap.cc:654] T d16a118ea4c04a58890c7dd3ff9ec7b4 P e4cfb8fc5a474177af6848a6efa13a27: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.009266 623 tablet_bootstrap.cc:492] T d16a118ea4c04a58890c7dd3ff9ec7b4 P e4cfb8fc5a474177af6848a6efa13a27: No bootstrap required, opened a new log
I20240502 14:15:18.012105 623 ts_tablet_manager.cc:1367] T d16a118ea4c04a58890c7dd3ff9ec7b4 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240502 14:15:18.018168 623 raft_consensus.cc:348] T d16a118ea4c04a58890c7dd3ff9ec7b4 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:18.020272 623 raft_consensus.cc:374] T d16a118ea4c04a58890c7dd3ff9ec7b4 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.020628 623 raft_consensus.cc:729] T d16a118ea4c04a58890c7dd3ff9ec7b4 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.021175 623 consensus_queue.cc:260] T d16a118ea4c04a58890c7dd3ff9ec7b4 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:18.024328 623 ts_tablet_manager.cc:1398] T d16a118ea4c04a58890c7dd3ff9ec7b4 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 14:15:18.037218 32232 raft_consensus.cc:482] T d16a118ea4c04a58890c7dd3ff9ec7b4 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:18.037427 32232 raft_consensus.cc:504] T d16a118ea4c04a58890c7dd3ff9ec7b4 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:18.038333 32232 leader_election.cc:288] T d16a118ea4c04a58890c7dd3ff9ec7b4 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:15:18.039198 26689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16a118ea4c04a58890c7dd3ff9ec7b4" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
W20240502 14:15:18.039855 26755 leader_election.cc:341] T d16a118ea4c04a58890c7dd3ff9ec7b4 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:18.041404 26423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16a118ea4c04a58890c7dd3ff9ec7b4" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
W20240502 14:15:18.042436 26754 leader_election.cc:341] T d16a118ea4c04a58890c7dd3ff9ec7b4 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:18.042599 26754 leader_election.cc:302] T d16a118ea4c04a58890c7dd3ff9ec7b4 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4cfb8fc5a474177af6848a6efa13a27; no voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, 89af7f423cf44717a08d6b142f0ed49d
I20240502 14:15:18.043215 324 raft_consensus.cc:2726] T d16a118ea4c04a58890c7dd3ff9ec7b4 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:15:18.085480 626 tablet_bootstrap.cc:492] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:15:18.089484 627 tablet_bootstrap.cc:492] T a42e68ab03f84d4b92893006fcf6f229 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:15:18.092921 627 tablet_bootstrap.cc:654] T a42e68ab03f84d4b92893006fcf6f229 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.095710 26284 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42842:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
W20240502 14:15:18.099285 29112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.196:32843, user_credentials={real_user=slave}} blocked reactor thread for 217876us
I20240502 14:15:18.106782 602 tablet.cc:1722] T c0d152eda1384b6792246b2f10d667da P 4092902b3e674857a0147f1b4f2cdb86: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:18.107347 602 tablet.cc:1620] T c0d152eda1384b6792246b2f10d667da P 4092902b3e674857a0147f1b4f2cdb86: MemRowSet was empty: no flush needed.
I20240502 14:15:18.110855 626 tablet_bootstrap.cc:654] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.116847 603 tablet.cc:1722] T c0d152eda1384b6792246b2f10d667da P ba187e38a2754d60a12de9ac4856430a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:18.117535 603 tablet.cc:1620] T c0d152eda1384b6792246b2f10d667da P ba187e38a2754d60a12de9ac4856430a: MemRowSet was empty: no flush needed.
I20240502 14:15:18.120260 26281 catalog_manager.cc:5576] TS 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781) does not have the latest schema for tablet c0d152eda1384b6792246b2f10d667da (table rebalancer_dynamic_table [id=f9780b3608294a1380a60ae3416d3829]). Expected version 2 got 1
I20240502 14:15:18.128515 26281 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09bae9f1528d44c09ade01fa361cebd0: rebalancer_dynamic_table [id=f9780b3608294a1380a60ae3416d3829] alter complete (version 2)
I20240502 14:15:18.160770 624 tablet_bootstrap.cc:492] T 46aaa6186d254ff7bd493bd85479015d P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:15:18.160974 621 tablet_bootstrap.cc:492] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3: No bootstrap required, opened a new log
I20240502 14:15:18.161177 621 ts_tablet_manager.cc:1367] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.176s user 0.012s sys 0.051s
I20240502 14:15:18.162827 624 tablet_bootstrap.cc:654] T 46aaa6186d254ff7bd493bd85479015d P c1100e16b3b9495ca61d4f95e89b6a89: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.163409 627 tablet_bootstrap.cc:492] T a42e68ab03f84d4b92893006fcf6f229 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:15:18.163647 627 ts_tablet_manager.cc:1367] T a42e68ab03f84d4b92893006fcf6f229 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.074s user 0.004s sys 0.000s
I20240502 14:15:18.164534 627 raft_consensus.cc:348] T a42e68ab03f84d4b92893006fcf6f229 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:18.164795 627 raft_consensus.cc:374] T a42e68ab03f84d4b92893006fcf6f229 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.164980 627 raft_consensus.cc:729] T a42e68ab03f84d4b92893006fcf6f229 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.165268 627 consensus_queue.cc:260] T a42e68ab03f84d4b92893006fcf6f229 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:18.163551 621 raft_consensus.cc:348] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:18.167325 621 raft_consensus.cc:374] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.167306 627 ts_tablet_manager.cc:1398] T a42e68ab03f84d4b92893006fcf6f229 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:15:18.167603 627 tablet_bootstrap.cc:492] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:15:18.167627 621 raft_consensus.cc:729] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.169018 604 tablet.cc:1722] T c0d152eda1384b6792246b2f10d667da P 68a3c2b17ed94e06aba66ed4178132ab: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:18.171825 604 tablet.cc:1620] T c0d152eda1384b6792246b2f10d667da P 68a3c2b17ed94e06aba66ed4178132ab: MemRowSet was empty: no flush needed.
I20240502 14:15:18.174162 621 consensus_queue.cc:260] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:18.175451 621 ts_tablet_manager.cc:1398] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.014s user 0.000s sys 0.003s
I20240502 14:15:18.180907 626 tablet_bootstrap.cc:492] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:15:18.181272 626 ts_tablet_manager.cc:1367] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.096s user 0.003s sys 0.001s
I20240502 14:15:18.182744 626 raft_consensus.cc:348] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:18.183251 626 raft_consensus.cc:374] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.183454 626 raft_consensus.cc:729] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.183665 624 tablet_bootstrap.cc:492] T 46aaa6186d254ff7bd493bd85479015d P c1100e16b3b9495ca61d4f95e89b6a89: No bootstrap required, opened a new log
I20240502 14:15:18.183871 626 consensus_queue.cc:260] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:18.185195 624 ts_tablet_manager.cc:1367] T 46aaa6186d254ff7bd493bd85479015d P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.000s
I20240502 14:15:18.186105 626 ts_tablet_manager.cc:1398] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 14:15:18.186579 626 tablet_bootstrap.cc:492] T 46aaa6186d254ff7bd493bd85479015d P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:15:18.189918 627 tablet_bootstrap.cc:654] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 89af7f423cf44717a08d6b142f0ed49d: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.192072 324 raft_consensus.cc:482] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:18.192258 324 raft_consensus.cc:504] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:18.192921 324 leader_election.cc:288] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:15:18.193920 29175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a42e68ab03f84d4b92893006fcf6f229" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:15:18.194180 29175 raft_consensus.cc:2454] T a42e68ab03f84d4b92893006fcf6f229 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 0.
I20240502 14:15:18.194720 26423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a42e68ab03f84d4b92893006fcf6f229" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:15:18.194934 26423 raft_consensus.cc:2454] T a42e68ab03f84d4b92893006fcf6f229 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 0.
I20240502 14:15:18.194932 26754 leader_election.cc:302] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:15:18.195648 324 raft_consensus.cc:2781] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:18.195747 324 raft_consensus.cc:482] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:18.195814 324 raft_consensus.cc:3037] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:18.196368 626 tablet_bootstrap.cc:654] T 46aaa6186d254ff7bd493bd85479015d P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.198781 324 raft_consensus.cc:504] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:18.199383 324 leader_election.cc:288] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Requested vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:15:18.201673 26423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a42e68ab03f84d4b92893006fcf6f229" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:15:18.201789 29175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a42e68ab03f84d4b92893006fcf6f229" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:15:18.201915 26423 raft_consensus.cc:3037] T a42e68ab03f84d4b92893006fcf6f229 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:18.202008 29175 raft_consensus.cc:3037] T a42e68ab03f84d4b92893006fcf6f229 P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:18.205226 29175 raft_consensus.cc:2454] T a42e68ab03f84d4b92893006fcf6f229 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
I20240502 14:15:18.205662 26754 leader_election.cc:302] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:15:18.205737 627 tablet_bootstrap.cc:492] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 89af7f423cf44717a08d6b142f0ed49d: No bootstrap required, opened a new log
I20240502 14:15:18.205838 626 tablet_bootstrap.cc:492] T 46aaa6186d254ff7bd493bd85479015d P 69ab78b2a1b74d2bbf2cdeafa0a4af23: No bootstrap required, opened a new log
I20240502 14:15:18.205973 626 ts_tablet_manager.cc:1367] T 46aaa6186d254ff7bd493bd85479015d P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.003s
I20240502 14:15:18.206219 32232 raft_consensus.cc:2781] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:18.206326 32232 raft_consensus.cc:686] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Becoming Leader. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Running, Role: LEADER
I20240502 14:15:18.206595 32232 consensus_queue.cc:237] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:18.208839 26423 raft_consensus.cc:2454] T a42e68ab03f84d4b92893006fcf6f229 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
I20240502 14:15:18.210247 624 raft_consensus.cc:348] T 46aaa6186d254ff7bd493bd85479015d P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:18.210536 624 raft_consensus.cc:374] T 46aaa6186d254ff7bd493bd85479015d P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.210570 627 ts_tablet_manager.cc:1367] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.002s
I20240502 14:15:18.210687 624 raft_consensus.cc:729] T 46aaa6186d254ff7bd493bd85479015d P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.211354 626 raft_consensus.cc:348] T 46aaa6186d254ff7bd493bd85479015d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:18.211383 624 consensus_queue.cc:260] T 46aaa6186d254ff7bd493bd85479015d P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:18.211546 626 raft_consensus.cc:374] T 46aaa6186d254ff7bd493bd85479015d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.211603 626 raft_consensus.cc:729] T 46aaa6186d254ff7bd493bd85479015d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.211747 627 raft_consensus.cc:348] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:18.212563 627 raft_consensus.cc:374] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.212764 627 raft_consensus.cc:729] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.213161 627 consensus_queue.cc:260] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:18.215068 626 consensus_queue.cc:260] T 46aaa6186d254ff7bd493bd85479015d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:18.215353 627 ts_tablet_manager.cc:1398] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240502 14:15:18.218223 626 ts_tablet_manager.cc:1398] T 46aaa6186d254ff7bd493bd85479015d P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240502 14:15:18.219086 624 ts_tablet_manager.cc:1398] T 46aaa6186d254ff7bd493bd85479015d P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.033s user 0.002s sys 0.000s
I20240502 14:15:18.225252 26285 catalog_manager.cc:5484] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196). New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:18.274560 411 raft_consensus.cc:482] T 46aaa6186d254ff7bd493bd85479015d P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:18.274749 411 raft_consensus.cc:504] T 46aaa6186d254ff7bd493bd85479015d P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:18.275647 411 leader_election.cc:288] T 46aaa6186d254ff7bd493bd85479015d P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:15:18.276460 26687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46aaa6186d254ff7bd493bd85479015d" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:15:18.276711 26687 raft_consensus.cc:2454] T 46aaa6186d254ff7bd493bd85479015d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 0.
I20240502 14:15:18.277269 29245 leader_election.cc:302] T 46aaa6186d254ff7bd493bd85479015d P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:15:18.277756 411 raft_consensus.cc:2781] T 46aaa6186d254ff7bd493bd85479015d P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:18.277901 411 raft_consensus.cc:482] T 46aaa6186d254ff7bd493bd85479015d P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:18.278015 411 raft_consensus.cc:3037] T 46aaa6186d254ff7bd493bd85479015d P c1100e16b3b9495ca61d4f95e89b6a89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:18.278975 29175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46aaa6186d254ff7bd493bd85479015d" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:15:18.279218 29175 raft_consensus.cc:2454] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 0.
I20240502 14:15:18.282094 411 raft_consensus.cc:504] T 46aaa6186d254ff7bd493bd85479015d P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:18.283094 411 leader_election.cc:288] T 46aaa6186d254ff7bd493bd85479015d P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 election: Requested vote from peers 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:15:18.283479 26687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46aaa6186d254ff7bd493bd85479015d" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:15:18.283665 26687 raft_consensus.cc:3037] T 46aaa6186d254ff7bd493bd85479015d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:18.283834 29174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46aaa6186d254ff7bd493bd85479015d" candidate_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:15:18.284112 29174 raft_consensus.cc:3037] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:18.294231 26687 raft_consensus.cc:2454] T 46aaa6186d254ff7bd493bd85479015d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 1.
I20240502 14:15:18.294469 29174 raft_consensus.cc:2454] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1100e16b3b9495ca61d4f95e89b6a89 in term 1.
I20240502 14:15:18.294986 29245 leader_election.cc:302] T 46aaa6186d254ff7bd493bd85479015d P c1100e16b3b9495ca61d4f95e89b6a89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:15:18.295423 411 raft_consensus.cc:2781] T 46aaa6186d254ff7bd493bd85479015d P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:18.295596 411 raft_consensus.cc:686] T 46aaa6186d254ff7bd493bd85479015d P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 LEADER]: Becoming Leader. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Running, Role: LEADER
I20240502 14:15:18.295945 411 consensus_queue.cc:237] T 46aaa6186d254ff7bd493bd85479015d P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:18.309175 32584 raft_consensus.cc:482] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:18.309322 32584 raft_consensus.cc:504] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:18.310024 32584 leader_election.cc:288] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:15:18.310616 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16a118ea4c04a58890c7dd3ff9ec7b4" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:15:18.310753 26687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16a118ea4c04a58890c7dd3ff9ec7b4" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:15:18.310863 26818 raft_consensus.cc:2454] T d16a118ea4c04a58890c7dd3ff9ec7b4 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:15:18.311072 26687 raft_consensus.cc:2454] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 0.
I20240502 14:15:18.311503 26363 leader_election.cc:302] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:15:18.312034 32445 raft_consensus.cc:2781] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:18.312124 32445 raft_consensus.cc:482] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:18.312197 32445 raft_consensus.cc:3037] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 89af7f423cf44717a08d6b142f0ed49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:18.315240 32445 raft_consensus.cc:504] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:18.316010 32445 leader_election.cc:288] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Requested vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:15:18.316843 26687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16a118ea4c04a58890c7dd3ff9ec7b4" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:15:18.317054 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16a118ea4c04a58890c7dd3ff9ec7b4" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:15:18.317118 26687 raft_consensus.cc:3037] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:18.317245 26818 raft_consensus.cc:3037] T d16a118ea4c04a58890c7dd3ff9ec7b4 P e4cfb8fc5a474177af6848a6efa13a27 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:18.320314 26687 raft_consensus.cc:2454] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:15:18.320708 26818 raft_consensus.cc:2454] T d16a118ea4c04a58890c7dd3ff9ec7b4 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:15:18.320850 26363 leader_election.cc:302] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:15:18.321419 32584 raft_consensus.cc:2781] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:18.321569 32584 raft_consensus.cc:686] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Becoming Leader. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Running, Role: LEADER
I20240502 14:15:18.321810 32584 consensus_queue.cc:237] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:18.334661 26283 catalog_manager.cc:5484] T 46aaa6186d254ff7bd493bd85479015d P c1100e16b3b9495ca61d4f95e89b6a89 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198). New cstate: current_term: 1 leader_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:18.340370 26283 catalog_manager.cc:5484] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: term changed from 0 to 1, leader changed from <none> to 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193). New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:18.364265 29155 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:18.515450 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:18.536275 26538 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:15:18.556715 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:18.562599 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:18.608171 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:18.626101 29421 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:18.666142 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:18.707154 32584 consensus_queue.cc:1035] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:18.739338 324 consensus_queue.cc:1035] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:18.777181 32439 consensus_queue.cc:1035] T 46aaa6186d254ff7bd493bd85479015d P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:18.818961 324 consensus_queue.cc:1035] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:18.849716 32439 consensus_queue.cc:1035] T 46aaa6186d254ff7bd493bd85479015d P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:18.868299 32445 consensus_queue.cc:1035] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:19.084914 26555 consensus_queue.cc:237] T d1aa4f4be5894ec5a64275f1172e1f55 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:19.089279 26423 raft_consensus.cc:1261] T d1aa4f4be5894ec5a64275f1172e1f55 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:19.089360 29174 raft_consensus.cc:1261] T d1aa4f4be5894ec5a64275f1172e1f55 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:19.090909 32650 consensus_queue.cc:1035] T d1aa4f4be5894ec5a64275f1172e1f55 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:19.091972 32650 consensus_queue.cc:1035] T d1aa4f4be5894ec5a64275f1172e1f55 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:19.105257 32650 raft_consensus.cc:2932] T d1aa4f4be5894ec5a64275f1172e1f55 P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:19.109230 29175 raft_consensus.cc:2932] T d1aa4f4be5894ec5a64275f1172e1f55 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:19.121107 26423 raft_consensus.cc:2932] T d1aa4f4be5894ec5a64275f1172e1f55 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:19.121634 26280 catalog_manager.cc:5484] T d1aa4f4be5894ec5a64275f1172e1f55 P ba187e38a2754d60a12de9ac4856430a reported cstate change: config changed from index 4 to 5, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:19.143806 26798 tablet_service.cc:1514] Processing DeleteTablet for tablet d1aa4f4be5894ec5a64275f1172e1f55 with delete_type TABLET_DATA_TOMBSTONED (TS e4cfb8fc5a474177af6848a6efa13a27 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:56010
I20240502 14:15:19.144620 714 tablet_replica.cc:330] T d1aa4f4be5894ec5a64275f1172e1f55 P e4cfb8fc5a474177af6848a6efa13a27: stopping tablet replica
I20240502 14:15:19.147212 714 raft_consensus.cc:2229] T d1aa4f4be5894ec5a64275f1172e1f55 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:19.149205 714 raft_consensus.cc:2258] T d1aa4f4be5894ec5a64275f1172e1f55 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:19.150246 714 ts_tablet_manager.cc:1875] T d1aa4f4be5894ec5a64275f1172e1f55 P e4cfb8fc5a474177af6848a6efa13a27: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:19.164727 714 ts_tablet_manager.cc:1888] T d1aa4f4be5894ec5a64275f1172e1f55 P e4cfb8fc5a474177af6848a6efa13a27: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:15:19.165024 714 log.cc:1196] T d1aa4f4be5894ec5a64275f1172e1f55 P e4cfb8fc5a474177af6848a6efa13a27: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/wals/d1aa4f4be5894ec5a64275f1172e1f55
I20240502 14:15:19.166103 26272 catalog_manager.cc:4830] TS e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): tablet d1aa4f4be5894ec5a64275f1172e1f55 (table rebalance_test_table_11 [id=c7861c8f192f462e8ed2198ec9c0c5ea]) successfully deleted
I20240502 14:15:19.565984 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:19.567456 29155 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:19.608764 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:19.630859 29421 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:19.667173 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:19.699453 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:19.734791 26538 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:15:19.735816 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:19.867669 26423 consensus_queue.cc:237] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:19.869830 29307 raft_consensus.cc:1261] T bb3e4ec75888419aa4e065cf5513ae5b P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:19.870005 26819 raft_consensus.cc:1261] T bb3e4ec75888419aa4e065cf5513ae5b P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:19.870512 653 consensus_queue.cc:1035] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:19.871240 32584 consensus_queue.cc:1035] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:19.874778 653 raft_consensus.cc:2932] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:15:19.875996 26818 raft_consensus.cc:2932] T bb3e4ec75888419aa4e065cf5513ae5b P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:15:19.882668 26278 catalog_manager.cc:5484] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: config changed from index 4 to 5, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) evicted. New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:15:19.890614 29307 raft_consensus.cc:2932] T bb3e4ec75888419aa4e065cf5513ae5b P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:15:19.905498 26538 tablet_service.cc:1514] Processing DeleteTablet for tablet bb3e4ec75888419aa4e065cf5513ae5b with delete_type TABLET_DATA_TOMBSTONED (TS ba187e38a2754d60a12de9ac4856430a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51696
I20240502 14:15:19.908095 807 tablet_replica.cc:330] T bb3e4ec75888419aa4e065cf5513ae5b P ba187e38a2754d60a12de9ac4856430a: stopping tablet replica
I20240502 14:15:19.908330 807 raft_consensus.cc:2229] T bb3e4ec75888419aa4e065cf5513ae5b P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:19.908567 807 raft_consensus.cc:2258] T bb3e4ec75888419aa4e065cf5513ae5b P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:19.909269 807 ts_tablet_manager.cc:1875] T bb3e4ec75888419aa4e065cf5513ae5b P ba187e38a2754d60a12de9ac4856430a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:19.916519 807 ts_tablet_manager.cc:1888] T bb3e4ec75888419aa4e065cf5513ae5b P ba187e38a2754d60a12de9ac4856430a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:15:19.916649 807 log.cc:1196] T bb3e4ec75888419aa4e065cf5513ae5b P ba187e38a2754d60a12de9ac4856430a: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/bb3e4ec75888419aa4e065cf5513ae5b
I20240502 14:15:19.918265 26271 catalog_manager.cc:4830] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): tablet bb3e4ec75888419aa4e065cf5513ae5b (table rebalance_test_table_19 [id=d4a5cfed144a42db8322f6823ef341a8]) successfully deleted
I20240502 14:15:20.225685 29155 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:20.235491 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:20.277807 29421 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:20.287461 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:20.315743 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:20.371649 26538 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:15:20.410293 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:20.451535 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:20.570793 26426 consensus_queue.cc:237] T 058eec4efd564c348684857372c60127 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:20.573318 29574 raft_consensus.cc:1261] T 058eec4efd564c348684857372c60127 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:20.573341 26558 raft_consensus.cc:1261] T 058eec4efd564c348684857372c60127 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:20.573889 653 consensus_queue.cc:1035] T 058eec4efd564c348684857372c60127 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:20.574476 653 consensus_queue.cc:1035] T 058eec4efd564c348684857372c60127 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:20.577517 32584 raft_consensus.cc:2932] T 058eec4efd564c348684857372c60127 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:15:20.581080 26558 raft_consensus.cc:2932] T 058eec4efd564c348684857372c60127 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:15:20.581373 29574 raft_consensus.cc:2932] T 058eec4efd564c348684857372c60127 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:15:20.583734 26280 catalog_manager.cc:5484] T 058eec4efd564c348684857372c60127 P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: config changed from index 4 to 5, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:15:20.608184 26798 tablet_service.cc:1514] Processing DeleteTablet for tablet 058eec4efd564c348684857372c60127 with delete_type TABLET_DATA_TOMBSTONED (TS e4cfb8fc5a474177af6848a6efa13a27 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:56010
I20240502 14:15:20.612916 897 tablet_replica.cc:330] T 058eec4efd564c348684857372c60127 P e4cfb8fc5a474177af6848a6efa13a27: stopping tablet replica
I20240502 14:15:20.613214 897 raft_consensus.cc:2229] T 058eec4efd564c348684857372c60127 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:20.613584 897 raft_consensus.cc:2258] T 058eec4efd564c348684857372c60127 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:20.614595 897 ts_tablet_manager.cc:1875] T 058eec4efd564c348684857372c60127 P e4cfb8fc5a474177af6848a6efa13a27: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:20.622997 897 ts_tablet_manager.cc:1888] T 058eec4efd564c348684857372c60127 P e4cfb8fc5a474177af6848a6efa13a27: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:15:20.623124 897 log.cc:1196] T 058eec4efd564c348684857372c60127 P e4cfb8fc5a474177af6848a6efa13a27: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/wals/058eec4efd564c348684857372c60127
I20240502 14:15:20.624097 26272 catalog_manager.cc:4830] TS e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): tablet 058eec4efd564c348684857372c60127 (table rebalance_test_table_17 [id=796786e80d834485b2cee5a25161aa97]) successfully deleted
I20240502 14:15:20.928217 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:20.944005 29155 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:20.960496 29421 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:21.015170 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:21.027043 26538 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:15:21.060817 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:21.075335 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:21.104226 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:21.315740 26818 consensus_queue.cc:237] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:21.318215 29442 raft_consensus.cc:1261] T 533b42de2de14abc96c00a928d5deed1 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:21.318243 26423 raft_consensus.cc:1261] T 533b42de2de14abc96c00a928d5deed1 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:21.319206 324 consensus_queue.cc:1035] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:21.319638 32232 consensus_queue.cc:1035] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:21.322860 324 raft_consensus.cc:2932] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:15:21.323552 26423 raft_consensus.cc:2932] T 533b42de2de14abc96c00a928d5deed1 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:15:21.329548 26284 catalog_manager.cc:5484] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: config changed from index 4 to 5, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:15:21.336943 29442 raft_consensus.cc:2932] T 533b42de2de14abc96c00a928d5deed1 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:15:21.376530 26669 tablet_service.cc:1514] Processing DeleteTablet for tablet 533b42de2de14abc96c00a928d5deed1 with delete_type TABLET_DATA_TOMBSTONED (TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34150
I20240502 14:15:21.388797 991 tablet_replica.cc:330] T 533b42de2de14abc96c00a928d5deed1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: stopping tablet replica
I20240502 14:15:21.389084 991 raft_consensus.cc:2229] T 533b42de2de14abc96c00a928d5deed1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:21.389353 991 raft_consensus.cc:2258] T 533b42de2de14abc96c00a928d5deed1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:21.390295 991 ts_tablet_manager.cc:1875] T 533b42de2de14abc96c00a928d5deed1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:21.404764 991 ts_tablet_manager.cc:1888] T 533b42de2de14abc96c00a928d5deed1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:15:21.405020 991 log.cc:1196] T 533b42de2de14abc96c00a928d5deed1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/wals/533b42de2de14abc96c00a928d5deed1
I20240502 14:15:21.407318 26272 catalog_manager.cc:4830] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): tablet 533b42de2de14abc96c00a928d5deed1 (table rebalance_test_table_15 [id=ff3107a4d4c94012a52f0cd0ca9ef235]) successfully deleted
I20240502 14:15:21.748809 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:21.764699 29155 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:21.784721 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:21.787082 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:21.802121 29421 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:21.822722 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:21.854089 26538 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:15:21.859757 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 14:15:21.943908 29245 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.199:41781, user_credentials={real_user=slave}} blocked reactor thread for 52214us
I20240502 14:15:22.226832 26817 consensus_queue.cc:237] T 0d0298cfb57342f7b840a8656f452645 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:22.229620 29442 raft_consensus.cc:1261] T 0d0298cfb57342f7b840a8656f452645 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:22.229995 29574 raft_consensus.cc:1261] T 0d0298cfb57342f7b840a8656f452645 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:22.230294 981 consensus_queue.cc:1035] T 0d0298cfb57342f7b840a8656f452645 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:22.230911 324 consensus_queue.cc:1035] T 0d0298cfb57342f7b840a8656f452645 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:15:22.236217 26754 consensus_peers.cc:485] T 0d0298cfb57342f7b840a8656f452645 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): Couldn't send request to peer 89af7f423cf44717a08d6b142f0ed49d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0d0298cfb57342f7b840a8656f452645. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:22.237282 324 raft_consensus.cc:2932] T 0d0298cfb57342f7b840a8656f452645 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:22.246896 29440 raft_consensus.cc:2932] T 0d0298cfb57342f7b840a8656f452645 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:22.256687 26277 catalog_manager.cc:5484] T 0d0298cfb57342f7b840a8656f452645 P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: config changed from index -1 to 2, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:22.256067 29574 raft_consensus.cc:2932] T 0d0298cfb57342f7b840a8656f452645 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:22.674042 29155 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:22.683007 29421 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:22.708566 1139 ts_tablet_manager.cc:902] T 0d0298cfb57342f7b840a8656f452645 P 89af7f423cf44717a08d6b142f0ed49d: Initiating tablet copy from peer e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
W20240502 14:15:22.774672 29511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.199:41781, user_credentials={real_user=slave}} blocked reactor thread for 123212us
I20240502 14:15:22.783373 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:22.814931 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:22.854770 1139 tablet_copy_client.cc:308] T 0d0298cfb57342f7b840a8656f452645 P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.196:32843
I20240502 14:15:22.859025 26839 tablet_copy_service.cc:140] P e4cfb8fc5a474177af6848a6efa13a27: Received BeginTabletCopySession request for tablet 0d0298cfb57342f7b840a8656f452645 from peer 89af7f423cf44717a08d6b142f0ed49d ({username='slave'} at 127.19.94.193:51541)
I20240502 14:15:22.859246 26839 tablet_copy_service.cc:161] P e4cfb8fc5a474177af6848a6efa13a27: Beginning new tablet copy session on tablet 0d0298cfb57342f7b840a8656f452645 from peer 89af7f423cf44717a08d6b142f0ed49d at {username='slave'} at 127.19.94.193:51541: session id = 89af7f423cf44717a08d6b142f0ed49d-0d0298cfb57342f7b840a8656f452645
I20240502 14:15:22.862282 26839 tablet_copy_source_session.cc:205] T 0d0298cfb57342f7b840a8656f452645 P e4cfb8fc5a474177af6848a6efa13a27: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:22.866699 1139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d0298cfb57342f7b840a8656f452645. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:22.891986 1139 tablet_copy_client.cc:783] T 0d0298cfb57342f7b840a8656f452645 P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:22.892324 1139 tablet_copy_client.cc:647] T 0d0298cfb57342f7b840a8656f452645 P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:22.893900 1139 tablet_copy_client.cc:523] T 0d0298cfb57342f7b840a8656f452645 P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:22.898620 26538 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:15:22.899416 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:22.906845 1139 tablet_bootstrap.cc:492] T 0d0298cfb57342f7b840a8656f452645 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:15:22.916684 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:22.963502 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 14:15:22.989521 29111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.198:43659, user_credentials={real_user=slave}} blocked reactor thread for 67246.7us
I20240502 14:15:22.995648 1139 tablet_bootstrap.cc:492] T 0d0298cfb57342f7b840a8656f452645 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:22.996184 1139 tablet_bootstrap.cc:492] T 0d0298cfb57342f7b840a8656f452645 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap complete.
I20240502 14:15:22.996546 1139 ts_tablet_manager.cc:1367] T 0d0298cfb57342f7b840a8656f452645 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.090s user 0.024s sys 0.031s
I20240502 14:15:22.997795 1139 raft_consensus.cc:348] T 0d0298cfb57342f7b840a8656f452645 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:22.998114 1139 raft_consensus.cc:729] T 0d0298cfb57342f7b840a8656f452645 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:15:22.998384 1139 consensus_queue.cc:260] T 0d0298cfb57342f7b840a8656f452645 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:22.999262 1139 ts_tablet_manager.cc:1398] T 0d0298cfb57342f7b840a8656f452645 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:23.000180 26839 tablet_copy_service.cc:342] P e4cfb8fc5a474177af6848a6efa13a27: Request end of tablet copy session 89af7f423cf44717a08d6b142f0ed49d-0d0298cfb57342f7b840a8656f452645 received from {username='slave'} at 127.19.94.193:51541
I20240502 14:15:23.000341 26839 tablet_copy_service.cc:433] P e4cfb8fc5a474177af6848a6efa13a27: ending tablet copy session 89af7f423cf44717a08d6b142f0ed49d-0d0298cfb57342f7b840a8656f452645 on tablet 0d0298cfb57342f7b840a8656f452645 with peer 89af7f423cf44717a08d6b142f0ed49d
I20240502 14:15:23.079420 26426 raft_consensus.cc:1203] T 0d0298cfb57342f7b840a8656f452645 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:23.559545 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:23.563283 29155 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:23.573053 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:23.590966 29421 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:23.627265 26538 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:15:23.646534 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240502 14:15:23.674319 29112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.200:46097, user_credentials={real_user=slave}} blocked reactor thread for 44454.1us
I20240502 14:15:23.723203 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:23.750650 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:23.878815 29573 consensus_queue.cc:237] T f8923ed1ce94441a8eb303b6f7dd3acb P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:23.881559 29175 raft_consensus.cc:1261] T f8923ed1ce94441a8eb303b6f7dd3acb P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:23.881834 26689 raft_consensus.cc:1261] T f8923ed1ce94441a8eb303b6f7dd3acb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:23.882421 867 consensus_queue.cc:1035] T f8923ed1ce94441a8eb303b6f7dd3acb P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:23.883186 867 consensus_queue.cc:1035] T f8923ed1ce94441a8eb303b6f7dd3acb P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:23.890977 1262 raft_consensus.cc:2932] T f8923ed1ce94441a8eb303b6f7dd3acb P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:23.891944 29175 raft_consensus.cc:2932] T f8923ed1ce94441a8eb303b6f7dd3acb P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:23.895352 26689 raft_consensus.cc:2932] T f8923ed1ce94441a8eb303b6f7dd3acb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:23.898270 26284 catalog_manager.cc:5484] T f8923ed1ce94441a8eb303b6f7dd3acb P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:23.910969 29421 tablet_service.cc:1514] Processing DeleteTablet for tablet f8923ed1ce94441a8eb303b6f7dd3acb with delete_type TABLET_DATA_TOMBSTONED (TS 68a3c2b17ed94e06aba66ed4178132ab not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:40652
I20240502 14:15:23.911839 1274 tablet_replica.cc:330] T f8923ed1ce94441a8eb303b6f7dd3acb P 68a3c2b17ed94e06aba66ed4178132ab: stopping tablet replica
I20240502 14:15:23.913710 1274 raft_consensus.cc:2229] T f8923ed1ce94441a8eb303b6f7dd3acb P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:23.914062 1274 raft_consensus.cc:2258] T f8923ed1ce94441a8eb303b6f7dd3acb P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:23.914860 1274 ts_tablet_manager.cc:1875] T f8923ed1ce94441a8eb303b6f7dd3acb P 68a3c2b17ed94e06aba66ed4178132ab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:23.922260 1274 ts_tablet_manager.cc:1888] T f8923ed1ce94441a8eb303b6f7dd3acb P 68a3c2b17ed94e06aba66ed4178132ab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:15:23.922431 1274 log.cc:1196] T f8923ed1ce94441a8eb303b6f7dd3acb P 68a3c2b17ed94e06aba66ed4178132ab: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal/wals/f8923ed1ce94441a8eb303b6f7dd3acb
I20240502 14:15:23.923333 26272 catalog_manager.cc:4830] TS 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): tablet f8923ed1ce94441a8eb303b6f7dd3acb (table rebalancer_extra_table_2 [id=df45ec00aaba41b2b3f62c6fb9633ae0]) successfully deleted
I20240502 14:15:24.429780 29155 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:24.433894 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:24.450846 29421 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:24.545800 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:24.565101 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:24.673655 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:24.714679 26538 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:15:24.767318 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:24.896766 26818 consensus_queue.cc:237] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:24.899104 26686 raft_consensus.cc:1261] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:24.899205 29440 raft_consensus.cc:1261] T ce03ee354c54457487ea939301ccd90e P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:24.899679 324 consensus_queue.cc:1035] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:24.900053 1255 consensus_queue.cc:1035] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:24.903921 324 raft_consensus.cc:2932] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:15:24.904614 26689 raft_consensus.cc:2932] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:15:24.907723 29440 raft_consensus.cc:2932] T ce03ee354c54457487ea939301ccd90e P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:15:24.915241 26283 catalog_manager.cc:5484] T ce03ee354c54457487ea939301ccd90e P 68a3c2b17ed94e06aba66ed4178132ab reported cstate change: config changed from index 4 to 5, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) evicted. New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:15:24.932461 26406 tablet_service.cc:1514] Processing DeleteTablet for tablet ce03ee354c54457487ea939301ccd90e with delete_type TABLET_DATA_TOMBSTONED (TS 89af7f423cf44717a08d6b142f0ed49d not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:37422
I20240502 14:15:24.937366 1391 tablet_replica.cc:330] T ce03ee354c54457487ea939301ccd90e P 89af7f423cf44717a08d6b142f0ed49d: stopping tablet replica
I20240502 14:15:24.937651 1391 raft_consensus.cc:2229] T ce03ee354c54457487ea939301ccd90e P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:24.937939 1391 raft_consensus.cc:2258] T ce03ee354c54457487ea939301ccd90e P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:24.938741 1391 ts_tablet_manager.cc:1875] T ce03ee354c54457487ea939301ccd90e P 89af7f423cf44717a08d6b142f0ed49d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:24.945380 1391 ts_tablet_manager.cc:1888] T ce03ee354c54457487ea939301ccd90e P 89af7f423cf44717a08d6b142f0ed49d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:15:24.945533 1391 log.cc:1196] T ce03ee354c54457487ea939301ccd90e P 89af7f423cf44717a08d6b142f0ed49d: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/ce03ee354c54457487ea939301ccd90e
I20240502 14:15:24.947345 26271 catalog_manager.cc:4830] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): tablet ce03ee354c54457487ea939301ccd90e (table rebalance_test_table_17 [id=796786e80d834485b2cee5a25161aa97]) successfully deleted
W20240502 14:15:25.257094 29511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.196:32843, user_credentials={real_user=slave}} blocked reactor thread for 56754.4us
I20240502 14:15:25.349567 29421 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:25.352093 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:25.375038 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:25.411603 26538 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:15:25.452229 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:25.530524 29155 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:25.568411 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:25.593253 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:25.741106 26818 consensus_queue.cc:237] T c586354e10cb4bedbbb20bd103ad477e P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:25.743309 29573 raft_consensus.cc:1261] T c586354e10cb4bedbbb20bd103ad477e P 4092902b3e674857a0147f1b4f2cdb86 [term 2 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:15:25.743299 26689 raft_consensus.cc:1261] T c586354e10cb4bedbbb20bd103ad477e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 2 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:15:25.743950 324 consensus_queue.cc:1035] T c586354e10cb4bedbbb20bd103ad477e P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:25.744300 1384 consensus_queue.cc:1035] T c586354e10cb4bedbbb20bd103ad477e P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:25.747471 324 raft_consensus.cc:2932] T c586354e10cb4bedbbb20bd103ad477e P e4cfb8fc5a474177af6848a6efa13a27 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:15:25.748272 26689 raft_consensus.cc:2932] T c586354e10cb4bedbbb20bd103ad477e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:15:25.750280 29573 raft_consensus.cc:2932] T c586354e10cb4bedbbb20bd103ad477e P 4092902b3e674857a0147f1b4f2cdb86 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:15:25.755180 26276 catalog_manager.cc:5484] T c586354e10cb4bedbbb20bd103ad477e P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: config changed from index 4 to 6, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) evicted. New cstate: current_term: 2 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:15:25.787725 26538 tablet_service.cc:1514] Processing DeleteTablet for tablet c586354e10cb4bedbbb20bd103ad477e with delete_type TABLET_DATA_TOMBSTONED (TS ba187e38a2754d60a12de9ac4856430a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51696
I20240502 14:15:25.788606 1488 tablet_replica.cc:330] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a: stopping tablet replica
I20240502 14:15:25.788887 1488 raft_consensus.cc:2229] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:25.789333 1488 raft_consensus.cc:2258] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:25.790033 1488 ts_tablet_manager.cc:1875] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:25.796458 1488 ts_tablet_manager.cc:1888] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 14:15:25.796730 1488 log.cc:1196] T c586354e10cb4bedbbb20bd103ad477e P ba187e38a2754d60a12de9ac4856430a: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/c586354e10cb4bedbbb20bd103ad477e
I20240502 14:15:25.797654 26271 catalog_manager.cc:4830] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): tablet c586354e10cb4bedbbb20bd103ad477e (table rebalance_test_table_3 [id=e814fcdd8ac94d6eb54d1c57f004c870]) successfully deleted
I20240502 14:15:25.919705 29573 consensus_queue.cc:237] T 5808efbc3ef84f3b862045c67b30cd8b P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:25.922569 29176 raft_consensus.cc:1261] T 5808efbc3ef84f3b862045c67b30cd8b P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:25.923184 29309 raft_consensus.cc:1261] T 5808efbc3ef84f3b862045c67b30cd8b P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:15:25.923282 29511 consensus_peers.cc:485] T 5808efbc3ef84f3b862045c67b30cd8b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): Couldn't send request to peer e4cfb8fc5a474177af6848a6efa13a27. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5808efbc3ef84f3b862045c67b30cd8b. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:25.923821 1436 consensus_queue.cc:1035] T 5808efbc3ef84f3b862045c67b30cd8b P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:25.924443 1262 consensus_queue.cc:1035] T 5808efbc3ef84f3b862045c67b30cd8b P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:25.929927 1436 raft_consensus.cc:2932] T 5808efbc3ef84f3b862045c67b30cd8b P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:25.930871 29176 raft_consensus.cc:2932] T 5808efbc3ef84f3b862045c67b30cd8b P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:25.935315 29309 raft_consensus.cc:2932] T 5808efbc3ef84f3b862045c67b30cd8b P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:25.940070 26281 catalog_manager.cc:5484] T 5808efbc3ef84f3b862045c67b30cd8b P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: config changed from index -1 to 2, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:25.976202 26818 consensus_queue.cc:237] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:25.978929 26689 raft_consensus.cc:1261] T 39a61744642f40c8ae92da708dc91ca3 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:25.979454 26423 raft_consensus.cc:1261] T 39a61744642f40c8ae92da708dc91ca3 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:25.979605 1384 consensus_queue.cc:1035] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:25.980024 324 consensus_queue.cc:1035] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:15:25.980715 26754 consensus_peers.cc:485] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): Couldn't send request to peer 1cb67158133e4648a6eceba32d4dbba3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39a61744642f40c8ae92da708dc91ca3. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:25.985378 1384 raft_consensus.cc:2932] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:25.989153 26423 raft_consensus.cc:2932] T 39a61744642f40c8ae92da708dc91ca3 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:25.990162 26689 raft_consensus.cc:2932] T 39a61744642f40c8ae92da708dc91ca3 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:25.991715 26276 catalog_manager.cc:5484] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: config changed from index -1 to 4, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:26.068747 29309 consensus_queue.cc:237] T abdd0f01d8d5453793c7e9461af10516 P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
W20240502 14:15:26.072371 29244 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P c1100e16b3b9495ca61d4f95e89b6a89 -> Peer ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): Couldn't send request to peer ba187e38a2754d60a12de9ac4856430a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: abdd0f01d8d5453793c7e9461af10516. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:26.072708 29440 raft_consensus.cc:1261] T abdd0f01d8d5453793c7e9461af10516 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer c1100e16b3b9495ca61d4f95e89b6a89: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:26.074036 1457 consensus_queue.cc:1035] T abdd0f01d8d5453793c7e9461af10516 P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:26.075578 29573 raft_consensus.cc:1261] T abdd0f01d8d5453793c7e9461af10516 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Refusing update from remote peer c1100e16b3b9495ca61d4f95e89b6a89: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:26.083796 1457 consensus_queue.cc:1035] T abdd0f01d8d5453793c7e9461af10516 P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20240502 14:15:26.089219 1457 raft_consensus.cc:2932] T abdd0f01d8d5453793c7e9461af10516 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:26.092545 29575 raft_consensus.cc:2932] T abdd0f01d8d5453793c7e9461af10516 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:26.097110 29440 raft_consensus.cc:2932] T abdd0f01d8d5453793c7e9461af10516 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:26.097561 26276 catalog_manager.cc:5484] T abdd0f01d8d5453793c7e9461af10516 P c1100e16b3b9495ca61d4f95e89b6a89 reported cstate change: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New cstate: current_term: 1 leader_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:26.171415 29176 consensus_queue.cc:237] T 39e6a71b3eef40848ea1ba4ef9604fa2 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:26.174690 26555 raft_consensus.cc:1261] T 39e6a71b3eef40848ea1ba4ef9604fa2 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:26.174782 29309 raft_consensus.cc:1261] T 39e6a71b3eef40848ea1ba4ef9604fa2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:26.175470 1238 consensus_queue.cc:1035] T 39e6a71b3eef40848ea1ba4ef9604fa2 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:26.175830 1604 consensus_queue.cc:1035] T 39e6a71b3eef40848ea1ba4ef9604fa2 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:15:26.175915 29112 consensus_peers.cc:485] T 39e6a71b3eef40848ea1ba4ef9604fa2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): Couldn't send request to peer e4cfb8fc5a474177af6848a6efa13a27. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e6a71b3eef40848ea1ba4ef9604fa2. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:26.182368 1238 raft_consensus.cc:2932] T 39e6a71b3eef40848ea1ba4ef9604fa2 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:26.185317 26555 raft_consensus.cc:2932] T 39e6a71b3eef40848ea1ba4ef9604fa2 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:26.187855 29309 raft_consensus.cc:2932] T 39e6a71b3eef40848ea1ba4ef9604fa2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:26.190503 26285 catalog_manager.cc:5484] T 39e6a71b3eef40848ea1ba4ef9604fa2 P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: config changed from index -1 to 2, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New cstate: current_term: 1 leader_uuid: "1cb67158133e4648a6eceba32d4dbba3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:26.252686 29575 consensus_queue.cc:237] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:26.255584 29440 raft_consensus.cc:1261] T 0c7ccae60f58427fbd39c26b98817963 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:15:26.256301 29510 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): Couldn't send request to peer 89af7f423cf44717a08d6b142f0ed49d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0c7ccae60f58427fbd39c26b98817963. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:26.256881 1262 consensus_queue.cc:1035] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:26.260473 26686 raft_consensus.cc:1261] T 0c7ccae60f58427fbd39c26b98817963 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:26.261431 1436 consensus_queue.cc:1035] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:26.270191 1262 raft_consensus.cc:2932] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:26.270987 26686 raft_consensus.cc:2932] T 0c7ccae60f58427fbd39c26b98817963 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:26.277974 26285 catalog_manager.cc:5484] T 0c7ccae60f58427fbd39c26b98817963 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: config changed from index -1 to 2, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:26.285635 29440 raft_consensus.cc:2932] T 0c7ccae60f58427fbd39c26b98817963 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:26.313877 1631 ts_tablet_manager.cc:902] T 5808efbc3ef84f3b862045c67b30cd8b P e4cfb8fc5a474177af6848a6efa13a27: Initiating tablet copy from peer 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:15:26.325361 1631 tablet_copy_client.cc:308] T 5808efbc3ef84f3b862045c67b30cd8b P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.200:46097
I20240502 14:15:26.326440 29594 tablet_copy_service.cc:140] P 4092902b3e674857a0147f1b4f2cdb86: Received BeginTabletCopySession request for tablet 5808efbc3ef84f3b862045c67b30cd8b from peer e4cfb8fc5a474177af6848a6efa13a27 ({username='slave'} at 127.19.94.196:45221)
I20240502 14:15:26.326694 29594 tablet_copy_service.cc:161] P 4092902b3e674857a0147f1b4f2cdb86: Beginning new tablet copy session on tablet 5808efbc3ef84f3b862045c67b30cd8b from peer e4cfb8fc5a474177af6848a6efa13a27 at {username='slave'} at 127.19.94.196:45221: session id = e4cfb8fc5a474177af6848a6efa13a27-5808efbc3ef84f3b862045c67b30cd8b
I20240502 14:15:26.329610 29594 tablet_copy_source_session.cc:205] T 5808efbc3ef84f3b862045c67b30cd8b P 4092902b3e674857a0147f1b4f2cdb86: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:26.332597 1631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5808efbc3ef84f3b862045c67b30cd8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:26.340250 1631 tablet_copy_client.cc:783] T 5808efbc3ef84f3b862045c67b30cd8b P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:26.340837 1631 tablet_copy_client.cc:647] T 5808efbc3ef84f3b862045c67b30cd8b P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:26.345736 1631 tablet_copy_client.cc:523] T 5808efbc3ef84f3b862045c67b30cd8b P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:26.349877 1631 tablet_bootstrap.cc:492] T 5808efbc3ef84f3b862045c67b30cd8b P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:15:26.358798 1631 tablet_bootstrap.cc:492] T 5808efbc3ef84f3b862045c67b30cd8b P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:26.359282 1631 tablet_bootstrap.cc:492] T 5808efbc3ef84f3b862045c67b30cd8b P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap complete.
I20240502 14:15:26.359586 1631 ts_tablet_manager.cc:1367] T 5808efbc3ef84f3b862045c67b30cd8b P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20240502 14:15:26.360632 1631 raft_consensus.cc:348] T 5808efbc3ef84f3b862045c67b30cd8b P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:26.360937 1631 raft_consensus.cc:729] T 5808efbc3ef84f3b862045c67b30cd8b P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:15:26.363037 1631 consensus_queue.cc:260] T 5808efbc3ef84f3b862045c67b30cd8b P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:26.363920 1631 ts_tablet_manager.cc:1398] T 5808efbc3ef84f3b862045c67b30cd8b P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:15:26.365106 29594 tablet_copy_service.cc:342] P 4092902b3e674857a0147f1b4f2cdb86: Request end of tablet copy session e4cfb8fc5a474177af6848a6efa13a27-5808efbc3ef84f3b862045c67b30cd8b received from {username='slave'} at 127.19.94.196:45221
I20240502 14:15:26.365252 29594 tablet_copy_service.cc:433] P 4092902b3e674857a0147f1b4f2cdb86: ending tablet copy session e4cfb8fc5a474177af6848a6efa13a27-5808efbc3ef84f3b862045c67b30cd8b on tablet 5808efbc3ef84f3b862045c67b30cd8b with peer e4cfb8fc5a474177af6848a6efa13a27
I20240502 14:15:26.375082 29176 consensus_queue.cc:237] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:26.378321 29440 raft_consensus.cc:1261] T f94a6ca24bf748c1ac4f12df761d537d P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:26.379155 29309 raft_consensus.cc:1261] T f94a6ca24bf748c1ac4f12df761d537d P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:26.380230 1460 consensus_queue.cc:1035] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:26.380947 1460 consensus_queue.cc:1035] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:15:26.383080 29111 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): Couldn't send request to peer ba187e38a2754d60a12de9ac4856430a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f94a6ca24bf748c1ac4f12df761d537d. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:26.393888 1460 raft_consensus.cc:2932] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:26.394629 29440 raft_consensus.cc:2932] T f94a6ca24bf748c1ac4f12df761d537d P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:26.402278 26276 catalog_manager.cc:5484] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New cstate: current_term: 1 leader_uuid: "1cb67158133e4648a6eceba32d4dbba3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:26.407527 29307 raft_consensus.cc:2932] T f94a6ca24bf748c1ac4f12df761d537d P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:26.470841 1666 ts_tablet_manager.cc:902] T abdd0f01d8d5453793c7e9461af10516 P ba187e38a2754d60a12de9ac4856430a: Initiating tablet copy from peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:15:26.474843 29440 consensus_queue.cc:237] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
W20240502 14:15:26.487732 29378 consensus_peers.cc:485] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b8430c13474b4850b71020c4a89bea68. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:26.488157 29575 raft_consensus.cc:1261] T b8430c13474b4850b71020c4a89bea68 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:26.489215 1585 consensus_queue.cc:1035] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:26.491053 26558 raft_consensus.cc:1261] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:26.491626 1332 consensus_queue.cc:1035] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:26.495155 1332 raft_consensus.cc:2932] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:15:26.499063 26558 raft_consensus.cc:2932] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:15:26.500015 29575 raft_consensus.cc:2932] T b8430c13474b4850b71020c4a89bea68 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:15:26.516856 1666 tablet_copy_client.cc:308] T abdd0f01d8d5453793c7e9461af10516 P ba187e38a2754d60a12de9ac4856430a: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.198:43659
I20240502 14:15:26.520315 29329 tablet_copy_service.cc:140] P c1100e16b3b9495ca61d4f95e89b6a89: Received BeginTabletCopySession request for tablet abdd0f01d8d5453793c7e9461af10516 from peer ba187e38a2754d60a12de9ac4856430a ({username='slave'} at 127.19.94.194:42791)
I20240502 14:15:26.520650 29329 tablet_copy_service.cc:161] P c1100e16b3b9495ca61d4f95e89b6a89: Beginning new tablet copy session on tablet abdd0f01d8d5453793c7e9461af10516 from peer ba187e38a2754d60a12de9ac4856430a at {username='slave'} at 127.19.94.194:42791: session id = ba187e38a2754d60a12de9ac4856430a-abdd0f01d8d5453793c7e9461af10516
I20240502 14:15:26.521844 26285 catalog_manager.cc:5484] T b8430c13474b4850b71020c4a89bea68 P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: config changed from index -1 to 2, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New cstate: current_term: 1 leader_uuid: "68a3c2b17ed94e06aba66ed4178132ab" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:15:26.525892 29329 tablet_copy_source_session.cc:205] T abdd0f01d8d5453793c7e9461af10516 P c1100e16b3b9495ca61d4f95e89b6a89: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:26.529865 1666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdd0f01d8d5453793c7e9461af10516. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:26.561828 1666 tablet_copy_client.cc:783] T abdd0f01d8d5453793c7e9461af10516 P ba187e38a2754d60a12de9ac4856430a: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:26.562134 1666 tablet_copy_client.cc:647] T abdd0f01d8d5453793c7e9461af10516 P ba187e38a2754d60a12de9ac4856430a: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:26.571336 1677 ts_tablet_manager.cc:902] T 39a61744642f40c8ae92da708dc91ca3 P 1cb67158133e4648a6eceba32d4dbba3: Initiating tablet copy from peer e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:15:26.573791 1666 tablet_copy_client.cc:523] T abdd0f01d8d5453793c7e9461af10516 P ba187e38a2754d60a12de9ac4856430a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:26.577620 1666 tablet_bootstrap.cc:492] T abdd0f01d8d5453793c7e9461af10516 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:15:26.577682 1677 tablet_copy_client.cc:308] T 39a61744642f40c8ae92da708dc91ca3 P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.196:32843
I20240502 14:15:26.578728 26839 tablet_copy_service.cc:140] P e4cfb8fc5a474177af6848a6efa13a27: Received BeginTabletCopySession request for tablet 39a61744642f40c8ae92da708dc91ca3 from peer 1cb67158133e4648a6eceba32d4dbba3 ({username='slave'} at 127.19.94.197:46803)
I20240502 14:15:26.578935 26839 tablet_copy_service.cc:161] P e4cfb8fc5a474177af6848a6efa13a27: Beginning new tablet copy session on tablet 39a61744642f40c8ae92da708dc91ca3 from peer 1cb67158133e4648a6eceba32d4dbba3 at {username='slave'} at 127.19.94.197:46803: session id = 1cb67158133e4648a6eceba32d4dbba3-39a61744642f40c8ae92da708dc91ca3
I20240502 14:15:26.581969 26839 tablet_copy_source_session.cc:205] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:26.585516 1677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39a61744642f40c8ae92da708dc91ca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:26.590461 1677 tablet_copy_client.cc:783] T 39a61744642f40c8ae92da708dc91ca3 P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:26.590754 1677 tablet_copy_client.cc:647] T 39a61744642f40c8ae92da708dc91ca3 P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:26.592729 1677 tablet_copy_client.cc:523] T 39a61744642f40c8ae92da708dc91ca3 P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:26.598984 1666 tablet_bootstrap.cc:492] T abdd0f01d8d5453793c7e9461af10516 P ba187e38a2754d60a12de9ac4856430a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:26.599434 1666 tablet_bootstrap.cc:492] T abdd0f01d8d5453793c7e9461af10516 P ba187e38a2754d60a12de9ac4856430a: Bootstrap complete.
I20240502 14:15:26.599697 1666 ts_tablet_manager.cc:1367] T abdd0f01d8d5453793c7e9461af10516 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.000s
I20240502 14:15:26.600005 1677 tablet_bootstrap.cc:492] T 39a61744642f40c8ae92da708dc91ca3 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:15:26.600533 1666 raft_consensus.cc:348] T abdd0f01d8d5453793c7e9461af10516 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:26.600769 1666 raft_consensus.cc:729] T abdd0f01d8d5453793c7e9461af10516 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:15:26.601123 1666 consensus_queue.cc:260] T abdd0f01d8d5453793c7e9461af10516 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:26.601729 1666 ts_tablet_manager.cc:1398] T abdd0f01d8d5453793c7e9461af10516 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:26.606875 29329 tablet_copy_service.cc:342] P c1100e16b3b9495ca61d4f95e89b6a89: Request end of tablet copy session ba187e38a2754d60a12de9ac4856430a-abdd0f01d8d5453793c7e9461af10516 received from {username='slave'} at 127.19.94.194:42791
I20240502 14:15:26.607079 29329 tablet_copy_service.cc:433] P c1100e16b3b9495ca61d4f95e89b6a89: ending tablet copy session ba187e38a2754d60a12de9ac4856430a-abdd0f01d8d5453793c7e9461af10516 on tablet abdd0f01d8d5453793c7e9461af10516 with peer ba187e38a2754d60a12de9ac4856430a
I20240502 14:15:26.645843 26558 consensus_queue.cc:237] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
W20240502 14:15:26.648509 26494 consensus_peers.cc:485] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a -> Peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): Couldn't send request to peer 69ab78b2a1b74d2bbf2cdeafa0a4af23. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 04f3eff43b544f33b020bd74e4331dea. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:26.648815 29575 raft_consensus.cc:1261] T 04f3eff43b544f33b020bd74e4331dea P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:26.649318 26817 raft_consensus.cc:1261] T 04f3eff43b544f33b020bd74e4331dea P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:26.649374 1679 consensus_queue.cc:1035] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:26.651886 1679 consensus_queue.cc:1035] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:26.657382 1631 ts_tablet_manager.cc:902] T 39e6a71b3eef40848ea1ba4ef9604fa2 P e4cfb8fc5a474177af6848a6efa13a27: Initiating tablet copy from peer 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:15:26.660929 1677 tablet_bootstrap.cc:492] T 39a61744642f40c8ae92da708dc91ca3 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:26.661396 1677 tablet_bootstrap.cc:492] T 39a61744642f40c8ae92da708dc91ca3 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap complete.
I20240502 14:15:26.661684 1677 ts_tablet_manager.cc:1367] T 39a61744642f40c8ae92da708dc91ca3 P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.062s user 0.036s sys 0.012s
I20240502 14:15:26.662406 1677 raft_consensus.cc:348] T 39a61744642f40c8ae92da708dc91ca3 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:26.662628 1677 raft_consensus.cc:729] T 39a61744642f40c8ae92da708dc91ca3 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:15:26.662878 1677 consensus_queue.cc:260] T 39a61744642f40c8ae92da708dc91ca3 P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:26.664691 1677 ts_tablet_manager.cc:1398] T 39a61744642f40c8ae92da708dc91ca3 P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:15:26.665400 26839 tablet_copy_service.cc:342] P e4cfb8fc5a474177af6848a6efa13a27: Request end of tablet copy session 1cb67158133e4648a6eceba32d4dbba3-39a61744642f40c8ae92da708dc91ca3 received from {username='slave'} at 127.19.94.197:46803
I20240502 14:15:26.665526 26839 tablet_copy_service.cc:433] P e4cfb8fc5a474177af6848a6efa13a27: ending tablet copy session 1cb67158133e4648a6eceba32d4dbba3-39a61744642f40c8ae92da708dc91ca3 on tablet 39a61744642f40c8ae92da708dc91ca3 with peer 1cb67158133e4648a6eceba32d4dbba3
I20240502 14:15:26.676777 1631 tablet_copy_client.cc:308] T 39e6a71b3eef40848ea1ba4ef9604fa2 P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.197:38715
I20240502 14:15:26.677780 29196 tablet_copy_service.cc:140] P 1cb67158133e4648a6eceba32d4dbba3: Received BeginTabletCopySession request for tablet 39e6a71b3eef40848ea1ba4ef9604fa2 from peer e4cfb8fc5a474177af6848a6efa13a27 ({username='slave'} at 127.19.94.196:53409)
I20240502 14:15:26.678026 29196 tablet_copy_service.cc:161] P 1cb67158133e4648a6eceba32d4dbba3: Beginning new tablet copy session on tablet 39e6a71b3eef40848ea1ba4ef9604fa2 from peer e4cfb8fc5a474177af6848a6efa13a27 at {username='slave'} at 127.19.94.196:53409: session id = e4cfb8fc5a474177af6848a6efa13a27-39e6a71b3eef40848ea1ba4ef9604fa2
I20240502 14:15:26.679035 1359 raft_consensus.cc:2932] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:26.679785 29574 raft_consensus.cc:2932] T 04f3eff43b544f33b020bd74e4331dea P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:26.681154 29196 tablet_copy_source_session.cc:205] T 39e6a71b3eef40848ea1ba4ef9604fa2 P 1cb67158133e4648a6eceba32d4dbba3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:26.682675 1631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39e6a71b3eef40848ea1ba4ef9604fa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:26.690017 26817 raft_consensus.cc:2932] T 04f3eff43b544f33b020bd74e4331dea P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:26.690227 26276 catalog_manager.cc:5484] T 04f3eff43b544f33b020bd74e4331dea P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:26.691573 1631 tablet_copy_client.cc:783] T 39e6a71b3eef40848ea1ba4ef9604fa2 P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:26.691828 1631 tablet_copy_client.cc:647] T 39e6a71b3eef40848ea1ba4ef9604fa2 P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:26.703514 1631 tablet_copy_client.cc:523] T 39e6a71b3eef40848ea1ba4ef9604fa2 P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:26.706815 1631 tablet_bootstrap.cc:492] T 39e6a71b3eef40848ea1ba4ef9604fa2 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:15:26.712985 1631 tablet_bootstrap.cc:492] T 39e6a71b3eef40848ea1ba4ef9604fa2 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:26.713685 1631 tablet_bootstrap.cc:492] T 39e6a71b3eef40848ea1ba4ef9604fa2 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap complete.
I20240502 14:15:26.714112 1631 ts_tablet_manager.cc:1367] T 39e6a71b3eef40848ea1ba4ef9604fa2 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.002s
I20240502 14:15:26.715281 1631 raft_consensus.cc:348] T 39e6a71b3eef40848ea1ba4ef9604fa2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:26.720213 1631 raft_consensus.cc:729] T 39e6a71b3eef40848ea1ba4ef9604fa2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:15:26.720623 1631 consensus_queue.cc:260] T 39e6a71b3eef40848ea1ba4ef9604fa2 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:26.721563 1631 ts_tablet_manager.cc:1398] T 39e6a71b3eef40848ea1ba4ef9604fa2 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:15:26.722823 29196 tablet_copy_service.cc:342] P 1cb67158133e4648a6eceba32d4dbba3: Request end of tablet copy session e4cfb8fc5a474177af6848a6efa13a27-39e6a71b3eef40848ea1ba4ef9604fa2 received from {username='slave'} at 127.19.94.196:53409
I20240502 14:15:26.723042 29196 tablet_copy_service.cc:433] P 1cb67158133e4648a6eceba32d4dbba3: ending tablet copy session e4cfb8fc5a474177af6848a6efa13a27-39e6a71b3eef40848ea1ba4ef9604fa2 on tablet 39e6a71b3eef40848ea1ba4ef9604fa2 with peer e4cfb8fc5a474177af6848a6efa13a27
I20240502 14:15:26.771589 26817 raft_consensus.cc:1203] T 5808efbc3ef84f3b862045c67b30cd8b P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:26.784621 29309 consensus_queue.cc:237] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:26.787761 29440 raft_consensus.cc:1261] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer c1100e16b3b9495ca61d4f95e89b6a89: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:15:26.788008 29244 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P c1100e16b3b9495ca61d4f95e89b6a89 -> Peer 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): Couldn't send request to peer 89af7f423cf44717a08d6b142f0ed49d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 842d4e7e5ee74e309fb7eba6d7b7ae8b. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:26.788148 26558 raft_consensus.cc:1261] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer c1100e16b3b9495ca61d4f95e89b6a89: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:26.788512 1457 consensus_queue.cc:1035] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:26.788921 1583 consensus_queue.cc:1035] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:26.792831 1583 raft_consensus.cc:2932] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:26.800230 26558 raft_consensus.cc:2932] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:26.808343 29442 raft_consensus.cc:2932] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:26.809505 26285 catalog_manager.cc:5484] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a reported cstate change: config changed from index -1 to 2, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New cstate: current_term: 1 leader_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:26.838482 1736 ts_tablet_manager.cc:902] T 0c7ccae60f58427fbd39c26b98817963 P 89af7f423cf44717a08d6b142f0ed49d: Initiating tablet copy from peer 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:15:26.842859 1736 tablet_copy_client.cc:308] T 0c7ccae60f58427fbd39c26b98817963 P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.200:46097
I20240502 14:15:26.844050 29594 tablet_copy_service.cc:140] P 4092902b3e674857a0147f1b4f2cdb86: Received BeginTabletCopySession request for tablet 0c7ccae60f58427fbd39c26b98817963 from peer 89af7f423cf44717a08d6b142f0ed49d ({username='slave'} at 127.19.94.193:43933)
I20240502 14:15:26.844377 29594 tablet_copy_service.cc:161] P 4092902b3e674857a0147f1b4f2cdb86: Beginning new tablet copy session on tablet 0c7ccae60f58427fbd39c26b98817963 from peer 89af7f423cf44717a08d6b142f0ed49d at {username='slave'} at 127.19.94.193:43933: session id = 89af7f423cf44717a08d6b142f0ed49d-0c7ccae60f58427fbd39c26b98817963
I20240502 14:15:26.849465 29594 tablet_copy_source_session.cc:205] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:26.852917 1736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c7ccae60f58427fbd39c26b98817963. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:26.861006 1736 tablet_copy_client.cc:783] T 0c7ccae60f58427fbd39c26b98817963 P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:26.861289 1736 tablet_copy_client.cc:647] T 0c7ccae60f58427fbd39c26b98817963 P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:26.863477 1736 tablet_copy_client.cc:523] T 0c7ccae60f58427fbd39c26b98817963 P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:26.866726 1736 tablet_bootstrap.cc:492] T 0c7ccae60f58427fbd39c26b98817963 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:15:26.874053 1736 tablet_bootstrap.cc:492] T 0c7ccae60f58427fbd39c26b98817963 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:26.874497 1736 tablet_bootstrap.cc:492] T 0c7ccae60f58427fbd39c26b98817963 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap complete.
I20240502 14:15:26.874769 1736 ts_tablet_manager.cc:1367] T 0c7ccae60f58427fbd39c26b98817963 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240502 14:15:26.875510 1736 raft_consensus.cc:348] T 0c7ccae60f58427fbd39c26b98817963 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:26.875741 1736 raft_consensus.cc:729] T 0c7ccae60f58427fbd39c26b98817963 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:15:26.875988 1736 consensus_queue.cc:260] T 0c7ccae60f58427fbd39c26b98817963 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:26.876595 1736 ts_tablet_manager.cc:1398] T 0c7ccae60f58427fbd39c26b98817963 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 14:15:26.877368 29594 tablet_copy_service.cc:342] P 4092902b3e674857a0147f1b4f2cdb86: Request end of tablet copy session 89af7f423cf44717a08d6b142f0ed49d-0c7ccae60f58427fbd39c26b98817963 received from {username='slave'} at 127.19.94.193:43933
I20240502 14:15:26.877617 29594 tablet_copy_service.cc:433] P 4092902b3e674857a0147f1b4f2cdb86: ending tablet copy session 89af7f423cf44717a08d6b142f0ed49d-0c7ccae60f58427fbd39c26b98817963 on tablet 0c7ccae60f58427fbd39c26b98817963 with peer 89af7f423cf44717a08d6b142f0ed49d
I20240502 14:15:26.900369 26818 consensus_queue.cc:237] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:26.904281 26423 raft_consensus.cc:1261] T a42e68ab03f84d4b92893006fcf6f229 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:26.904632 29176 raft_consensus.cc:1261] T a42e68ab03f84d4b92893006fcf6f229 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:15:26.904845 26754 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): Couldn't send request to peer ba187e38a2754d60a12de9ac4856430a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a42e68ab03f84d4b92893006fcf6f229. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:26.905586 1384 consensus_queue.cc:1035] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:26.906286 1384 consensus_queue.cc:1035] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:26.918900 26558 raft_consensus.cc:1203] T abdd0f01d8d5453793c7e9461af10516 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:26.919132 1384 raft_consensus.cc:2932] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:26.921475 26423 raft_consensus.cc:2932] T a42e68ab03f84d4b92893006fcf6f229 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:26.924749 1666 ts_tablet_manager.cc:902] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a: Initiating tablet copy from peer 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:15:26.932327 26285 catalog_manager.cc:5484] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:26.933334 29176 raft_consensus.cc:2932] T a42e68ab03f84d4b92893006fcf6f229 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:26.972663 1666 tablet_copy_client.cc:308] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.197:38715
I20240502 14:15:26.973549 29196 tablet_copy_service.cc:140] P 1cb67158133e4648a6eceba32d4dbba3: Received BeginTabletCopySession request for tablet f94a6ca24bf748c1ac4f12df761d537d from peer ba187e38a2754d60a12de9ac4856430a ({username='slave'} at 127.19.94.194:50529)
I20240502 14:15:26.973766 29196 tablet_copy_service.cc:161] P 1cb67158133e4648a6eceba32d4dbba3: Beginning new tablet copy session on tablet f94a6ca24bf748c1ac4f12df761d537d from peer ba187e38a2754d60a12de9ac4856430a at {username='slave'} at 127.19.94.194:50529: session id = ba187e38a2754d60a12de9ac4856430a-f94a6ca24bf748c1ac4f12df761d537d
I20240502 14:15:26.976572 29196 tablet_copy_source_session.cc:205] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:26.978667 1666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f94a6ca24bf748c1ac4f12df761d537d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:26.984928 1666 tablet_copy_client.cc:783] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:26.985216 1666 tablet_copy_client.cc:647] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:26.990058 1666 tablet_copy_client.cc:523] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:26.993526 26423 consensus_queue.cc:237] T 2e682a5c24a64fe3ab7d325a027242d9 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:26.993816 1666 tablet_bootstrap.cc:492] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:15:26.997586 29309 raft_consensus.cc:1261] T 2e682a5c24a64fe3ab7d325a027242d9 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:26.997731 26686 raft_consensus.cc:1261] T 2e682a5c24a64fe3ab7d325a027242d9 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:26.998232 1297 consensus_queue.cc:1035] T 2e682a5c24a64fe3ab7d325a027242d9 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:15:26.999071 26363 consensus_peers.cc:485] T 2e682a5c24a64fe3ab7d325a027242d9 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097): Couldn't send request to peer 4092902b3e674857a0147f1b4f2cdb86. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e682a5c24a64fe3ab7d325a027242d9. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:26.999086 888 consensus_queue.cc:1035] T 2e682a5c24a64fe3ab7d325a027242d9 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:27.006736 1666 tablet_bootstrap.cc:492] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:27.007061 1666 tablet_bootstrap.cc:492] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a: Bootstrap complete.
I20240502 14:15:27.007272 1666 ts_tablet_manager.cc:1367] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20240502 14:15:27.007886 1666 raft_consensus.cc:348] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:27.008056 1666 raft_consensus.cc:729] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:15:27.008268 1666 consensus_queue.cc:260] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:27.009126 1666 ts_tablet_manager.cc:1398] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240502 14:15:27.010210 29196 tablet_copy_service.cc:342] P 1cb67158133e4648a6eceba32d4dbba3: Request end of tablet copy session ba187e38a2754d60a12de9ac4856430a-f94a6ca24bf748c1ac4f12df761d537d received from {username='slave'} at 127.19.94.194:50529
I20240502 14:15:27.010354 29196 tablet_copy_service.cc:433] P 1cb67158133e4648a6eceba32d4dbba3: ending tablet copy session ba187e38a2754d60a12de9ac4856430a-f94a6ca24bf748c1ac4f12df761d537d on tablet f94a6ca24bf748c1ac4f12df761d537d with peer ba187e38a2754d60a12de9ac4856430a
I20240502 14:15:27.011548 888 raft_consensus.cc:2932] T 2e682a5c24a64fe3ab7d325a027242d9 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:15:27.014055 29309 raft_consensus.cc:2932] T 2e682a5c24a64fe3ab7d325a027242d9 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:15:27.023249 29176 raft_consensus.cc:1203] T 39a61744642f40c8ae92da708dc91ca3 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:15:27.028620 26285 catalog_manager.cc:5484] T 2e682a5c24a64fe3ab7d325a027242d9 P c1100e16b3b9495ca61d4f95e89b6a89 reported cstate change: config changed from index -1 to 2, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:15:27.036829 26686 raft_consensus.cc:2932] T 2e682a5c24a64fe3ab7d325a027242d9 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:15:27.082881 1790 ts_tablet_manager.cc:902] T b8430c13474b4850b71020c4a89bea68 P c1100e16b3b9495ca61d4f95e89b6a89: Initiating tablet copy from peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:15:27.148680 1790 tablet_copy_client.cc:308] T b8430c13474b4850b71020c4a89bea68 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.199:41781
I20240502 14:15:27.149536 29463 tablet_copy_service.cc:140] P 68a3c2b17ed94e06aba66ed4178132ab: Received BeginTabletCopySession request for tablet b8430c13474b4850b71020c4a89bea68 from peer c1100e16b3b9495ca61d4f95e89b6a89 ({username='slave'} at 127.19.94.198:50467)
I20240502 14:15:27.149761 29463 tablet_copy_service.cc:161] P 68a3c2b17ed94e06aba66ed4178132ab: Beginning new tablet copy session on tablet b8430c13474b4850b71020c4a89bea68 from peer c1100e16b3b9495ca61d4f95e89b6a89 at {username='slave'} at 127.19.94.198:50467: session id = c1100e16b3b9495ca61d4f95e89b6a89-b8430c13474b4850b71020c4a89bea68
I20240502 14:15:27.153704 29463 tablet_copy_source_session.cc:205] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:27.155704 1790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8430c13474b4850b71020c4a89bea68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:27.165630 1790 tablet_copy_client.cc:783] T b8430c13474b4850b71020c4a89bea68 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:27.165967 1790 tablet_copy_client.cc:647] T b8430c13474b4850b71020c4a89bea68 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:27.167847 1790 tablet_copy_client.cc:523] T b8430c13474b4850b71020c4a89bea68 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:27.171643 1790 tablet_bootstrap.cc:492] T b8430c13474b4850b71020c4a89bea68 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:15:27.183424 26689 consensus_queue.cc:237] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:27.185251 1790 tablet_bootstrap.cc:492] T b8430c13474b4850b71020c4a89bea68 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:27.185740 1790 tablet_bootstrap.cc:492] T b8430c13474b4850b71020c4a89bea68 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap complete.
I20240502 14:15:27.186069 1790 ts_tablet_manager.cc:1367] T b8430c13474b4850b71020c4a89bea68 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.007s
I20240502 14:15:27.187074 1790 raft_consensus.cc:348] T b8430c13474b4850b71020c4a89bea68 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:27.187312 1790 raft_consensus.cc:729] T b8430c13474b4850b71020c4a89bea68 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:15:27.187652 1790 consensus_queue.cc:260] T b8430c13474b4850b71020c4a89bea68 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:27.190131 1790 ts_tablet_manager.cc:1398] T b8430c13474b4850b71020c4a89bea68 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 14:15:27.190886 29463 tablet_copy_service.cc:342] P 68a3c2b17ed94e06aba66ed4178132ab: Request end of tablet copy session c1100e16b3b9495ca61d4f95e89b6a89-b8430c13474b4850b71020c4a89bea68 received from {username='slave'} at 127.19.94.198:50467
I20240502 14:15:27.191041 29463 tablet_copy_service.cc:433] P 68a3c2b17ed94e06aba66ed4178132ab: ending tablet copy session c1100e16b3b9495ca61d4f95e89b6a89-b8430c13474b4850b71020c4a89bea68 on tablet b8430c13474b4850b71020c4a89bea68 with peer c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:15:27.195359 1806 ts_tablet_manager.cc:902] T 04f3eff43b544f33b020bd74e4331dea P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Initiating tablet copy from peer ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:15:27.200498 26817 raft_consensus.cc:1203] T 39e6a71b3eef40848ea1ba4ef9604fa2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:27.205935 1736 ts_tablet_manager.cc:902] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 89af7f423cf44717a08d6b142f0ed49d: Initiating tablet copy from peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:15:27.216459 1806 tablet_copy_client.cc:308] T 04f3eff43b544f33b020bd74e4331dea P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.194:36025
W20240502 14:15:27.217288 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cf52d0f868fa45418af48639e12f3282. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:27.218484 26578 tablet_copy_service.cc:140] P ba187e38a2754d60a12de9ac4856430a: Received BeginTabletCopySession request for tablet 04f3eff43b544f33b020bd74e4331dea from peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 ({username='slave'} at 127.19.94.195:34463)
I20240502 14:15:27.218640 26578 tablet_copy_service.cc:161] P ba187e38a2754d60a12de9ac4856430a: Beginning new tablet copy session on tablet 04f3eff43b544f33b020bd74e4331dea from peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 at {username='slave'} at 127.19.94.195:34463: session id = 69ab78b2a1b74d2bbf2cdeafa0a4af23-04f3eff43b544f33b020bd74e4331dea
I20240502 14:15:27.219835 26558 raft_consensus.cc:1261] T cf52d0f868fa45418af48639e12f3282 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:27.220480 1752 consensus_queue.cc:1035] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:27.221992 26578 tablet_copy_source_session.cc:205] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:27.222890 26817 raft_consensus.cc:1261] T cf52d0f868fa45418af48639e12f3282 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:27.223762 1752 consensus_queue.cc:1035] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:27.228080 1806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04f3eff43b544f33b020bd74e4331dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:27.235183 1806 tablet_copy_client.cc:783] T 04f3eff43b544f33b020bd74e4331dea P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:27.235460 1806 tablet_copy_client.cc:647] T 04f3eff43b544f33b020bd74e4331dea P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:27.237414 1806 tablet_copy_client.cc:523] T 04f3eff43b544f33b020bd74e4331dea P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:27.237704 1022 raft_consensus.cc:2932] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:15:27.238406 26558 raft_consensus.cc:2932] T cf52d0f868fa45418af48639e12f3282 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:15:27.247457 26285 catalog_manager.cc:5484] T cf52d0f868fa45418af48639e12f3282 P ba187e38a2754d60a12de9ac4856430a reported cstate change: config changed from index -1 to 4, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:15:27.265309 26817 raft_consensus.cc:2932] T cf52d0f868fa45418af48639e12f3282 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:15:27.266556 1736 tablet_copy_client.cc:308] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.198:43659
I20240502 14:15:27.267344 29329 tablet_copy_service.cc:140] P c1100e16b3b9495ca61d4f95e89b6a89: Received BeginTabletCopySession request for tablet 842d4e7e5ee74e309fb7eba6d7b7ae8b from peer 89af7f423cf44717a08d6b142f0ed49d ({username='slave'} at 127.19.94.193:58815)
I20240502 14:15:27.267534 29329 tablet_copy_service.cc:161] P c1100e16b3b9495ca61d4f95e89b6a89: Beginning new tablet copy session on tablet 842d4e7e5ee74e309fb7eba6d7b7ae8b from peer 89af7f423cf44717a08d6b142f0ed49d at {username='slave'} at 127.19.94.193:58815: session id = 89af7f423cf44717a08d6b142f0ed49d-842d4e7e5ee74e309fb7eba6d7b7ae8b
I20240502 14:15:27.270505 29329 tablet_copy_source_session.cc:205] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P c1100e16b3b9495ca61d4f95e89b6a89: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:27.274631 1806 tablet_bootstrap.cc:492] T 04f3eff43b544f33b020bd74e4331dea P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:15:27.280069 1736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 842d4e7e5ee74e309fb7eba6d7b7ae8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:27.290135 1806 tablet_bootstrap.cc:492] T 04f3eff43b544f33b020bd74e4331dea P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:27.290536 1806 tablet_bootstrap.cc:492] T 04f3eff43b544f33b020bd74e4331dea P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap complete.
I20240502 14:15:27.290752 1806 ts_tablet_manager.cc:1367] T 04f3eff43b544f33b020bd74e4331dea P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.004s
I20240502 14:15:27.291321 1806 raft_consensus.cc:348] T 04f3eff43b544f33b020bd74e4331dea P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:27.291488 1806 raft_consensus.cc:729] T 04f3eff43b544f33b020bd74e4331dea P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:15:27.291707 1806 consensus_queue.cc:260] T 04f3eff43b544f33b020bd74e4331dea P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:27.293463 1806 ts_tablet_manager.cc:1398] T 04f3eff43b544f33b020bd74e4331dea P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 14:15:27.294332 26578 tablet_copy_service.cc:342] P ba187e38a2754d60a12de9ac4856430a: Request end of tablet copy session 69ab78b2a1b74d2bbf2cdeafa0a4af23-04f3eff43b544f33b020bd74e4331dea received from {username='slave'} at 127.19.94.195:34463
I20240502 14:15:27.294443 26578 tablet_copy_service.cc:433] P ba187e38a2754d60a12de9ac4856430a: ending tablet copy session 69ab78b2a1b74d2bbf2cdeafa0a4af23-04f3eff43b544f33b020bd74e4331dea on tablet 04f3eff43b544f33b020bd74e4331dea with peer 69ab78b2a1b74d2bbf2cdeafa0a4af23
I20240502 14:15:27.296504 1736 tablet_copy_client.cc:783] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:27.296744 1736 tablet_copy_client.cc:647] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:27.309990 1736 tablet_copy_client.cc:523] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:27.314850 1736 tablet_bootstrap.cc:492] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:15:27.334875 1736 tablet_bootstrap.cc:492] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:27.335299 1736 tablet_bootstrap.cc:492] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap complete.
I20240502 14:15:27.335556 1736 ts_tablet_manager.cc:1367] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.000s
I20240502 14:15:27.336704 1736 raft_consensus.cc:348] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:27.336941 1736 raft_consensus.cc:729] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:15:27.337229 1736 consensus_queue.cc:260] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:27.337896 1736 ts_tablet_manager.cc:1398] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 14:15:27.339447 26426 raft_consensus.cc:1203] T 0c7ccae60f58427fbd39c26b98817963 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:27.343590 29329 tablet_copy_service.cc:342] P c1100e16b3b9495ca61d4f95e89b6a89: Request end of tablet copy session 89af7f423cf44717a08d6b142f0ed49d-842d4e7e5ee74e309fb7eba6d7b7ae8b received from {username='slave'} at 127.19.94.193:58815
I20240502 14:15:27.343735 29329 tablet_copy_service.cc:433] P c1100e16b3b9495ca61d4f95e89b6a89: ending tablet copy session 89af7f423cf44717a08d6b142f0ed49d-842d4e7e5ee74e309fb7eba6d7b7ae8b on tablet 842d4e7e5ee74e309fb7eba6d7b7ae8b with peer 89af7f423cf44717a08d6b142f0ed49d
I20240502 14:15:27.407142 26426 consensus_queue.cc:237] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:27.409641 26689 raft_consensus.cc:1261] T 9f3cbedfa1da42698615a544d9dc3276 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:27.409662 26558 raft_consensus.cc:1261] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:27.410176 1783 consensus_queue.cc:1035] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:15:27.410465 26362 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): Couldn't send request to peer 1cb67158133e4648a6eceba32d4dbba3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9f3cbedfa1da42698615a544d9dc3276. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:27.410470 1297 consensus_queue.cc:1035] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:27.413115 1783 raft_consensus.cc:2932] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:27.413790 26689 raft_consensus.cc:2932] T 9f3cbedfa1da42698615a544d9dc3276 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:27.416728 26558 raft_consensus.cc:1203] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:27.419097 26555 raft_consensus.cc:2932] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:27.422195 26285 catalog_manager.cc:5484] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: config changed from index -1 to 4, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:27.437278 1845 ts_tablet_manager.cc:902] T 2e682a5c24a64fe3ab7d325a027242d9 P 4092902b3e674857a0147f1b4f2cdb86: Initiating tablet copy from peer 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:15:27.464319 1845 tablet_copy_client.cc:308] T 2e682a5c24a64fe3ab7d325a027242d9 P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.193:37985
I20240502 14:15:27.465322 26447 tablet_copy_service.cc:140] P 89af7f423cf44717a08d6b142f0ed49d: Received BeginTabletCopySession request for tablet 2e682a5c24a64fe3ab7d325a027242d9 from peer 4092902b3e674857a0147f1b4f2cdb86 ({username='slave'} at 127.19.94.200:60351)
I20240502 14:15:27.465497 26447 tablet_copy_service.cc:161] P 89af7f423cf44717a08d6b142f0ed49d: Beginning new tablet copy session on tablet 2e682a5c24a64fe3ab7d325a027242d9 from peer 4092902b3e674857a0147f1b4f2cdb86 at {username='slave'} at 127.19.94.200:60351: session id = 4092902b3e674857a0147f1b4f2cdb86-2e682a5c24a64fe3ab7d325a027242d9
I20240502 14:15:27.468294 26447 tablet_copy_source_session.cc:205] T 2e682a5c24a64fe3ab7d325a027242d9 P 89af7f423cf44717a08d6b142f0ed49d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:27.469786 1845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e682a5c24a64fe3ab7d325a027242d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:27.477862 1845 tablet_copy_client.cc:783] T 2e682a5c24a64fe3ab7d325a027242d9 P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:27.478204 1845 tablet_copy_client.cc:647] T 2e682a5c24a64fe3ab7d325a027242d9 P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:27.481961 1845 tablet_copy_client.cc:523] T 2e682a5c24a64fe3ab7d325a027242d9 P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:27.485918 1845 tablet_bootstrap.cc:492] T 2e682a5c24a64fe3ab7d325a027242d9 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:15:27.486822 1666 ts_tablet_manager.cc:902] T a42e68ab03f84d4b92893006fcf6f229 P ba187e38a2754d60a12de9ac4856430a: Initiating tablet copy from peer e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:15:27.488102 26689 consensus_queue.cc:237] T 0f2a4042e0f64ceebb345b9cf9764805 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
W20240502 14:15:27.500370 26624 consensus_peers.cc:485] T 0f2a4042e0f64ceebb345b9cf9764805 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): Couldn't send request to peer ba187e38a2754d60a12de9ac4856430a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f2a4042e0f64ceebb345b9cf9764805. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:27.504033 1666 tablet_copy_client.cc:308] T a42e68ab03f84d4b92893006fcf6f229 P ba187e38a2754d60a12de9ac4856430a: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.196:32843
I20240502 14:15:27.505362 26839 tablet_copy_service.cc:140] P e4cfb8fc5a474177af6848a6efa13a27: Received BeginTabletCopySession request for tablet a42e68ab03f84d4b92893006fcf6f229 from peer ba187e38a2754d60a12de9ac4856430a ({username='slave'} at 127.19.94.194:53513)
I20240502 14:15:27.505599 26839 tablet_copy_service.cc:161] P e4cfb8fc5a474177af6848a6efa13a27: Beginning new tablet copy session on tablet a42e68ab03f84d4b92893006fcf6f229 from peer ba187e38a2754d60a12de9ac4856430a at {username='slave'} at 127.19.94.194:53513: session id = ba187e38a2754d60a12de9ac4856430a-a42e68ab03f84d4b92893006fcf6f229
I20240502 14:15:27.507799 26818 raft_consensus.cc:1261] T 0f2a4042e0f64ceebb345b9cf9764805 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:27.507997 26426 raft_consensus.cc:1261] T 0f2a4042e0f64ceebb345b9cf9764805 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:27.509016 1022 consensus_queue.cc:1035] T 0f2a4042e0f64ceebb345b9cf9764805 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:27.509747 1022 consensus_queue.cc:1035] T 0f2a4042e0f64ceebb345b9cf9764805 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:27.510571 1845 tablet_bootstrap.cc:492] T 2e682a5c24a64fe3ab7d325a027242d9 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:27.511037 26839 tablet_copy_source_session.cc:205] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:27.511152 1845 tablet_bootstrap.cc:492] T 2e682a5c24a64fe3ab7d325a027242d9 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap complete.
I20240502 14:15:27.511464 1845 ts_tablet_manager.cc:1367] T 2e682a5c24a64fe3ab7d325a027242d9 P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.026s user 0.015s sys 0.006s
I20240502 14:15:27.512538 1666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a42e68ab03f84d4b92893006fcf6f229. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:27.512766 1845 raft_consensus.cc:348] T 2e682a5c24a64fe3ab7d325a027242d9 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:27.513054 1845 raft_consensus.cc:729] T 2e682a5c24a64fe3ab7d325a027242d9 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:15:27.513430 1845 consensus_queue.cc:260] T 2e682a5c24a64fe3ab7d325a027242d9 P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:27.513998 1845 ts_tablet_manager.cc:1398] T 2e682a5c24a64fe3ab7d325a027242d9 P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 14:15:27.514668 26447 tablet_copy_service.cc:342] P 89af7f423cf44717a08d6b142f0ed49d: Request end of tablet copy session 4092902b3e674857a0147f1b4f2cdb86-2e682a5c24a64fe3ab7d325a027242d9 received from {username='slave'} at 127.19.94.200:60351
I20240502 14:15:27.514778 26447 tablet_copy_service.cc:433] P 89af7f423cf44717a08d6b142f0ed49d: ending tablet copy session 4092902b3e674857a0147f1b4f2cdb86-2e682a5c24a64fe3ab7d325a027242d9 on tablet 2e682a5c24a64fe3ab7d325a027242d9 with peer 4092902b3e674857a0147f1b4f2cdb86
I20240502 14:15:27.518638 1666 tablet_copy_client.cc:783] T a42e68ab03f84d4b92893006fcf6f229 P ba187e38a2754d60a12de9ac4856430a: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:27.518883 1666 tablet_copy_client.cc:647] T a42e68ab03f84d4b92893006fcf6f229 P ba187e38a2754d60a12de9ac4856430a: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:27.521039 1022 raft_consensus.cc:2932] T 0f2a4042e0f64ceebb345b9cf9764805 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:27.525828 1666 tablet_copy_client.cc:523] T a42e68ab03f84d4b92893006fcf6f229 P ba187e38a2754d60a12de9ac4856430a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:27.525652 26426 raft_consensus.cc:2932] T 0f2a4042e0f64ceebb345b9cf9764805 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:27.529436 1666 tablet_bootstrap.cc:492] T a42e68ab03f84d4b92893006fcf6f229 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:15:27.533377 26818 raft_consensus.cc:2932] T 0f2a4042e0f64ceebb345b9cf9764805 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:27.536770 26285 catalog_manager.cc:5484] T 0f2a4042e0f64ceebb345b9cf9764805 P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:27.539848 1666 tablet_bootstrap.cc:492] T a42e68ab03f84d4b92893006fcf6f229 P ba187e38a2754d60a12de9ac4856430a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:27.540246 1666 tablet_bootstrap.cc:492] T a42e68ab03f84d4b92893006fcf6f229 P ba187e38a2754d60a12de9ac4856430a: Bootstrap complete.
I20240502 14:15:27.540529 1666 ts_tablet_manager.cc:1367] T a42e68ab03f84d4b92893006fcf6f229 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.006s
I20240502 14:15:27.541211 1666 raft_consensus.cc:348] T a42e68ab03f84d4b92893006fcf6f229 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:27.541412 1666 raft_consensus.cc:729] T a42e68ab03f84d4b92893006fcf6f229 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:15:27.541640 1666 consensus_queue.cc:260] T a42e68ab03f84d4b92893006fcf6f229 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:27.542263 1666 ts_tablet_manager.cc:1398] T a42e68ab03f84d4b92893006fcf6f229 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:15:27.543577 26839 tablet_copy_service.cc:342] P e4cfb8fc5a474177af6848a6efa13a27: Request end of tablet copy session ba187e38a2754d60a12de9ac4856430a-a42e68ab03f84d4b92893006fcf6f229 received from {username='slave'} at 127.19.94.194:53513
I20240502 14:15:27.543732 26839 tablet_copy_service.cc:433] P e4cfb8fc5a474177af6848a6efa13a27: ending tablet copy session ba187e38a2754d60a12de9ac4856430a-a42e68ab03f84d4b92893006fcf6f229 on tablet a42e68ab03f84d4b92893006fcf6f229 with peer ba187e38a2754d60a12de9ac4856430a
I20240502 14:15:27.591478 26686 raft_consensus.cc:1203] T 04f3eff43b544f33b020bd74e4331dea P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:27.597798 29575 consensus_queue.cc:237] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
W20240502 14:15:27.601356 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): Couldn't send request to peer e4cfb8fc5a474177af6848a6efa13a27. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9671eeb3b0e44730a4b0c2849e6ee012. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:27.601433 29440 raft_consensus.cc:1261] T 9671eeb3b0e44730a4b0c2849e6ee012 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:27.601501 29309 raft_consensus.cc:1261] T 9671eeb3b0e44730a4b0c2849e6ee012 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:27.602180 1545 consensus_queue.cc:1035] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:27.602780 1545 consensus_queue.cc:1035] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:27.614982 1625 raft_consensus.cc:2932] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:27.616534 29309 raft_consensus.cc:2932] T 9671eeb3b0e44730a4b0c2849e6ee012 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:27.621024 29440 raft_consensus.cc:2932] T 9671eeb3b0e44730a4b0c2849e6ee012 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:27.623473 26285 catalog_manager.cc:5484] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: config changed from index -1 to 2, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:27.670737 1901 ts_tablet_manager.cc:902] T cf52d0f868fa45418af48639e12f3282 P 68a3c2b17ed94e06aba66ed4178132ab: Initiating tablet copy from peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:15:27.678231 1901 tablet_copy_client.cc:308] T cf52d0f868fa45418af48639e12f3282 P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.195:44325
I20240502 14:15:27.679152 26709 tablet_copy_service.cc:140] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Received BeginTabletCopySession request for tablet cf52d0f868fa45418af48639e12f3282 from peer 68a3c2b17ed94e06aba66ed4178132ab ({username='slave'} at 127.19.94.199:51755)
I20240502 14:15:27.679395 26709 tablet_copy_service.cc:161] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Beginning new tablet copy session on tablet cf52d0f868fa45418af48639e12f3282 from peer 68a3c2b17ed94e06aba66ed4178132ab at {username='slave'} at 127.19.94.199:51755: session id = 68a3c2b17ed94e06aba66ed4178132ab-cf52d0f868fa45418af48639e12f3282
I20240502 14:15:27.683010 29309 raft_consensus.cc:1203] T b8430c13474b4850b71020c4a89bea68 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:27.687361 26709 tablet_copy_source_session.cc:205] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:27.689729 1901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf52d0f868fa45418af48639e12f3282. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:27.689561 26818 consensus_queue.cc:237] T a823ce0b7b174b53902a85b7f2e9f59f P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:27.692288 29574 raft_consensus.cc:1261] T a823ce0b7b174b53902a85b7f2e9f59f P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:27.692718 29176 raft_consensus.cc:1261] T a823ce0b7b174b53902a85b7f2e9f59f P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:27.693295 1384 consensus_queue.cc:1035] T a823ce0b7b174b53902a85b7f2e9f59f P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:15:27.693572 26755 consensus_peers.cc:485] T a823ce0b7b174b53902a85b7f2e9f59f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): Couldn't send request to peer 69ab78b2a1b74d2bbf2cdeafa0a4af23. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a823ce0b7b174b53902a85b7f2e9f59f. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:27.693871 1384 consensus_queue.cc:1035] T a823ce0b7b174b53902a85b7f2e9f59f P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:27.697762 1912 raft_consensus.cc:2932] T a823ce0b7b174b53902a85b7f2e9f59f P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:27.698269 29176 raft_consensus.cc:2932] T a823ce0b7b174b53902a85b7f2e9f59f P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:27.699494 1901 tablet_copy_client.cc:783] T cf52d0f868fa45418af48639e12f3282 P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:27.699784 1901 tablet_copy_client.cc:647] T cf52d0f868fa45418af48639e12f3282 P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:27.706426 26276 catalog_manager.cc:5484] T a823ce0b7b174b53902a85b7f2e9f59f P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:27.712852 29574 raft_consensus.cc:2932] T a823ce0b7b174b53902a85b7f2e9f59f P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:27.717489 1901 tablet_copy_client.cc:523] T cf52d0f868fa45418af48639e12f3282 P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:27.722819 1901 tablet_bootstrap.cc:492] T cf52d0f868fa45418af48639e12f3282 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:15:27.769582 26426 raft_consensus.cc:1203] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:27.788892 1901 tablet_bootstrap.cc:492] T cf52d0f868fa45418af48639e12f3282 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:27.789453 1901 tablet_bootstrap.cc:492] T cf52d0f868fa45418af48639e12f3282 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap complete.
I20240502 14:15:27.789765 1901 ts_tablet_manager.cc:1367] T cf52d0f868fa45418af48639e12f3282 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.067s user 0.047s sys 0.007s
I20240502 14:15:27.791817 1901 raft_consensus.cc:348] T cf52d0f868fa45418af48639e12f3282 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:27.792122 1901 raft_consensus.cc:729] T cf52d0f868fa45418af48639e12f3282 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:15:27.792358 1901 consensus_queue.cc:260] T cf52d0f868fa45418af48639e12f3282 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:27.793123 1901 ts_tablet_manager.cc:1398] T cf52d0f868fa45418af48639e12f3282 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 14:15:27.794068 26709 tablet_copy_service.cc:342] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Request end of tablet copy session 68a3c2b17ed94e06aba66ed4178132ab-cf52d0f868fa45418af48639e12f3282 received from {username='slave'} at 127.19.94.199:51755
I20240502 14:15:27.794237 26709 tablet_copy_service.cc:433] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: ending tablet copy session 68a3c2b17ed94e06aba66ed4178132ab-cf52d0f868fa45418af48639e12f3282 on tablet cf52d0f868fa45418af48639e12f3282 with peer 68a3c2b17ed94e06aba66ed4178132ab
I20240502 14:15:27.839226 1938 ts_tablet_manager.cc:902] T 9f3cbedfa1da42698615a544d9dc3276 P 1cb67158133e4648a6eceba32d4dbba3: Initiating tablet copy from peer 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:15:27.847184 1938 tablet_copy_client.cc:308] T 9f3cbedfa1da42698615a544d9dc3276 P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.193:37985
I20240502 14:15:27.847955 26447 tablet_copy_service.cc:140] P 89af7f423cf44717a08d6b142f0ed49d: Received BeginTabletCopySession request for tablet 9f3cbedfa1da42698615a544d9dc3276 from peer 1cb67158133e4648a6eceba32d4dbba3 ({username='slave'} at 127.19.94.197:56601)
I20240502 14:15:27.848125 26447 tablet_copy_service.cc:161] P 89af7f423cf44717a08d6b142f0ed49d: Beginning new tablet copy session on tablet 9f3cbedfa1da42698615a544d9dc3276 from peer 1cb67158133e4648a6eceba32d4dbba3 at {username='slave'} at 127.19.94.197:56601: session id = 1cb67158133e4648a6eceba32d4dbba3-9f3cbedfa1da42698615a544d9dc3276
I20240502 14:15:27.850338 26447 tablet_copy_source_session.cc:205] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:27.851668 1938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f3cbedfa1da42698615a544d9dc3276. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:27.857546 1938 tablet_copy_client.cc:783] T 9f3cbedfa1da42698615a544d9dc3276 P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:27.858052 1938 tablet_copy_client.cc:647] T 9f3cbedfa1da42698615a544d9dc3276 P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:27.864928 1938 tablet_copy_client.cc:523] T 9f3cbedfa1da42698615a544d9dc3276 P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:27.866705 26555 raft_consensus.cc:1203] T a42e68ab03f84d4b92893006fcf6f229 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:27.869439 1938 tablet_bootstrap.cc:492] T 9f3cbedfa1da42698615a544d9dc3276 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:15:27.941881 1938 tablet_bootstrap.cc:492] T 9f3cbedfa1da42698615a544d9dc3276 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:27.942453 1938 tablet_bootstrap.cc:492] T 9f3cbedfa1da42698615a544d9dc3276 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap complete.
I20240502 14:15:27.942797 1938 ts_tablet_manager.cc:1367] T 9f3cbedfa1da42698615a544d9dc3276 P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.073s user 0.050s sys 0.001s
I20240502 14:15:27.943853 1938 raft_consensus.cc:348] T 9f3cbedfa1da42698615a544d9dc3276 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:27.944114 1938 raft_consensus.cc:729] T 9f3cbedfa1da42698615a544d9dc3276 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:15:27.944520 1938 consensus_queue.cc:260] T 9f3cbedfa1da42698615a544d9dc3276 P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:27.945145 1938 ts_tablet_manager.cc:1398] T 9f3cbedfa1da42698615a544d9dc3276 P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 14:15:27.945868 26447 tablet_copy_service.cc:342] P 89af7f423cf44717a08d6b142f0ed49d: Request end of tablet copy session 1cb67158133e4648a6eceba32d4dbba3-9f3cbedfa1da42698615a544d9dc3276 received from {username='slave'} at 127.19.94.197:56601
I20240502 14:15:27.945986 26447 tablet_copy_service.cc:433] P 89af7f423cf44717a08d6b142f0ed49d: ending tablet copy session 1cb67158133e4648a6eceba32d4dbba3-9f3cbedfa1da42698615a544d9dc3276 on tablet 9f3cbedfa1da42698615a544d9dc3276 with peer 1cb67158133e4648a6eceba32d4dbba3
I20240502 14:15:28.006399 29574 raft_consensus.cc:1203] T 2e682a5c24a64fe3ab7d325a027242d9 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:28.017333 1967 ts_tablet_manager.cc:902] T 9671eeb3b0e44730a4b0c2849e6ee012 P e4cfb8fc5a474177af6848a6efa13a27: Initiating tablet copy from peer 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:15:28.054674 1967 tablet_copy_client.cc:308] T 9671eeb3b0e44730a4b0c2849e6ee012 P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.200:46097
I20240502 14:15:28.055562 29594 tablet_copy_service.cc:140] P 4092902b3e674857a0147f1b4f2cdb86: Received BeginTabletCopySession request for tablet 9671eeb3b0e44730a4b0c2849e6ee012 from peer e4cfb8fc5a474177af6848a6efa13a27 ({username='slave'} at 127.19.94.196:45221)
I20240502 14:15:28.055747 29594 tablet_copy_service.cc:161] P 4092902b3e674857a0147f1b4f2cdb86: Beginning new tablet copy session on tablet 9671eeb3b0e44730a4b0c2849e6ee012 from peer e4cfb8fc5a474177af6848a6efa13a27 at {username='slave'} at 127.19.94.196:45221: session id = e4cfb8fc5a474177af6848a6efa13a27-9671eeb3b0e44730a4b0c2849e6ee012
I20240502 14:15:28.058727 29594 tablet_copy_source_session.cc:205] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:28.060642 1967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9671eeb3b0e44730a4b0c2849e6ee012. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:28.070658 1967 tablet_copy_client.cc:783] T 9671eeb3b0e44730a4b0c2849e6ee012 P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:28.071029 1967 tablet_copy_client.cc:647] T 9671eeb3b0e44730a4b0c2849e6ee012 P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:28.075343 1967 tablet_copy_client.cc:523] T 9671eeb3b0e44730a4b0c2849e6ee012 P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:28.077776 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:28.084547 1967 tablet_bootstrap.cc:492] T 9671eeb3b0e44730a4b0c2849e6ee012 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:15:28.095392 29155 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:28.110555 29442 raft_consensus.cc:1203] T cf52d0f868fa45418af48639e12f3282 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:15:28.126495 29421 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:28.138325 1979 ts_tablet_manager.cc:902] T 0f2a4042e0f64ceebb345b9cf9764805 P ba187e38a2754d60a12de9ac4856430a: Initiating tablet copy from peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:15:28.164045 1967 tablet_bootstrap.cc:492] T 9671eeb3b0e44730a4b0c2849e6ee012 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:28.166417 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:28.174050 1967 tablet_bootstrap.cc:492] T 9671eeb3b0e44730a4b0c2849e6ee012 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap complete.
I20240502 14:15:28.174486 1967 ts_tablet_manager.cc:1367] T 9671eeb3b0e44730a4b0c2849e6ee012 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.090s user 0.015s sys 0.006s
I20240502 14:15:28.176357 1967 raft_consensus.cc:348] T 9671eeb3b0e44730a4b0c2849e6ee012 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:28.176659 1967 raft_consensus.cc:729] T 9671eeb3b0e44730a4b0c2849e6ee012 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:15:28.181106 1967 consensus_queue.cc:260] T 9671eeb3b0e44730a4b0c2849e6ee012 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:28.183997 1967 ts_tablet_manager.cc:1398] T 9671eeb3b0e44730a4b0c2849e6ee012 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240502 14:15:28.184844 29594 tablet_copy_service.cc:342] P 4092902b3e674857a0147f1b4f2cdb86: Request end of tablet copy session e4cfb8fc5a474177af6848a6efa13a27-9671eeb3b0e44730a4b0c2849e6ee012 received from {username='slave'} at 127.19.94.196:45221
I20240502 14:15:28.184986 29594 tablet_copy_service.cc:433] P 4092902b3e674857a0147f1b4f2cdb86: ending tablet copy session e4cfb8fc5a474177af6848a6efa13a27-9671eeb3b0e44730a4b0c2849e6ee012 on tablet 9671eeb3b0e44730a4b0c2849e6ee012 with peer e4cfb8fc5a474177af6848a6efa13a27
I20240502 14:15:28.201256 1979 tablet_copy_client.cc:308] T 0f2a4042e0f64ceebb345b9cf9764805 P ba187e38a2754d60a12de9ac4856430a: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.195:44325
I20240502 14:15:28.204617 26709 tablet_copy_service.cc:140] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Received BeginTabletCopySession request for tablet 0f2a4042e0f64ceebb345b9cf9764805 from peer ba187e38a2754d60a12de9ac4856430a ({username='slave'} at 127.19.94.194:58393)
I20240502 14:15:28.204821 26709 tablet_copy_service.cc:161] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Beginning new tablet copy session on tablet 0f2a4042e0f64ceebb345b9cf9764805 from peer ba187e38a2754d60a12de9ac4856430a at {username='slave'} at 127.19.94.194:58393: session id = ba187e38a2754d60a12de9ac4856430a-0f2a4042e0f64ceebb345b9cf9764805
I20240502 14:15:28.221336 29176 raft_consensus.cc:1203] T 9f3cbedfa1da42698615a544d9dc3276 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:15:28.207850 26709 tablet_copy_source_session.cc:205] T 0f2a4042e0f64ceebb345b9cf9764805 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:28.224184 1979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f2a4042e0f64ceebb345b9cf9764805. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:28.220121 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:28.264110 1979 tablet_copy_client.cc:783] T 0f2a4042e0f64ceebb345b9cf9764805 P ba187e38a2754d60a12de9ac4856430a: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:28.264948 1979 tablet_copy_client.cc:647] T 0f2a4042e0f64ceebb345b9cf9764805 P ba187e38a2754d60a12de9ac4856430a: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:28.267262 1979 tablet_copy_client.cc:523] T 0f2a4042e0f64ceebb345b9cf9764805 P ba187e38a2754d60a12de9ac4856430a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240502 14:15:28.275223 29378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.194:36025, user_credentials={real_user=slave}} blocked reactor thread for 54964.8us
I20240502 14:15:28.291092 1979 tablet_bootstrap.cc:492] T 0f2a4042e0f64ceebb345b9cf9764805 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:15:28.324481 1994 ts_tablet_manager.cc:902] T a823ce0b7b174b53902a85b7f2e9f59f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Initiating tablet copy from peer e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:15:28.344019 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:28.344842 1979 tablet_bootstrap.cc:492] T 0f2a4042e0f64ceebb345b9cf9764805 P ba187e38a2754d60a12de9ac4856430a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:28.345229 1979 tablet_bootstrap.cc:492] T 0f2a4042e0f64ceebb345b9cf9764805 P ba187e38a2754d60a12de9ac4856430a: Bootstrap complete.
I20240502 14:15:28.345499 1979 ts_tablet_manager.cc:1367] T 0f2a4042e0f64ceebb345b9cf9764805 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.055s user 0.007s sys 0.005s
I20240502 14:15:28.346351 1979 raft_consensus.cc:348] T 0f2a4042e0f64ceebb345b9cf9764805 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:28.346591 1979 raft_consensus.cc:729] T 0f2a4042e0f64ceebb345b9cf9764805 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:15:28.346818 1979 consensus_queue.cc:260] T 0f2a4042e0f64ceebb345b9cf9764805 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:28.364486 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:28.374418 1979 ts_tablet_manager.cc:1398] T 0f2a4042e0f64ceebb345b9cf9764805 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.029s user 0.003s sys 0.000s
I20240502 14:15:28.386296 26709 tablet_copy_service.cc:342] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Request end of tablet copy session ba187e38a2754d60a12de9ac4856430a-0f2a4042e0f64ceebb345b9cf9764805 received from {username='slave'} at 127.19.94.194:58393
I20240502 14:15:28.386441 26709 tablet_copy_service.cc:433] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: ending tablet copy session ba187e38a2754d60a12de9ac4856430a-0f2a4042e0f64ceebb345b9cf9764805 on tablet 0f2a4042e0f64ceebb345b9cf9764805 with peer ba187e38a2754d60a12de9ac4856430a
I20240502 14:15:28.401064 26538 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:15:28.459635 1994 tablet_copy_client.cc:308] T a823ce0b7b174b53902a85b7f2e9f59f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.196:32843
I20240502 14:15:28.460563 26839 tablet_copy_service.cc:140] P e4cfb8fc5a474177af6848a6efa13a27: Received BeginTabletCopySession request for tablet a823ce0b7b174b53902a85b7f2e9f59f from peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 ({username='slave'} at 127.19.94.195:42141)
I20240502 14:15:28.460727 26839 tablet_copy_service.cc:161] P e4cfb8fc5a474177af6848a6efa13a27: Beginning new tablet copy session on tablet a823ce0b7b174b53902a85b7f2e9f59f from peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 at {username='slave'} at 127.19.94.195:42141: session id = 69ab78b2a1b74d2bbf2cdeafa0a4af23-a823ce0b7b174b53902a85b7f2e9f59f
I20240502 14:15:28.463742 26839 tablet_copy_source_session.cc:205] T a823ce0b7b174b53902a85b7f2e9f59f P e4cfb8fc5a474177af6848a6efa13a27: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:28.465342 1994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a823ce0b7b174b53902a85b7f2e9f59f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:28.465972 26555 raft_consensus.cc:1203] T 0f2a4042e0f64ceebb345b9cf9764805 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:28.475080 1994 tablet_copy_client.cc:783] T a823ce0b7b174b53902a85b7f2e9f59f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:28.475356 1994 tablet_copy_client.cc:647] T a823ce0b7b174b53902a85b7f2e9f59f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:28.477155 1994 tablet_copy_client.cc:523] T a823ce0b7b174b53902a85b7f2e9f59f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:28.481202 1994 tablet_bootstrap.cc:492] T a823ce0b7b174b53902a85b7f2e9f59f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:15:28.523553 1994 tablet_bootstrap.cc:492] T a823ce0b7b174b53902a85b7f2e9f59f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:28.523936 1994 tablet_bootstrap.cc:492] T a823ce0b7b174b53902a85b7f2e9f59f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap complete.
I20240502 14:15:28.525327 1994 ts_tablet_manager.cc:1367] T a823ce0b7b174b53902a85b7f2e9f59f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.044s user 0.014s sys 0.023s
I20240502 14:15:28.526015 1994 raft_consensus.cc:348] T a823ce0b7b174b53902a85b7f2e9f59f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:28.526302 1994 raft_consensus.cc:729] T a823ce0b7b174b53902a85b7f2e9f59f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:15:28.526556 1994 consensus_queue.cc:260] T a823ce0b7b174b53902a85b7f2e9f59f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:28.527201 1994 ts_tablet_manager.cc:1398] T a823ce0b7b174b53902a85b7f2e9f59f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:15:28.527957 26839 tablet_copy_service.cc:342] P e4cfb8fc5a474177af6848a6efa13a27: Request end of tablet copy session 69ab78b2a1b74d2bbf2cdeafa0a4af23-a823ce0b7b174b53902a85b7f2e9f59f received from {username='slave'} at 127.19.94.195:42141
I20240502 14:15:28.528081 26839 tablet_copy_service.cc:433] P e4cfb8fc5a474177af6848a6efa13a27: ending tablet copy session 69ab78b2a1b74d2bbf2cdeafa0a4af23-a823ce0b7b174b53902a85b7f2e9f59f on tablet a823ce0b7b174b53902a85b7f2e9f59f with peer 69ab78b2a1b74d2bbf2cdeafa0a4af23
I20240502 14:15:28.614904 26817 raft_consensus.cc:1203] T 9671eeb3b0e44730a4b0c2849e6ee012 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:28.820858 26689 raft_consensus.cc:1203] T a823ce0b7b174b53902a85b7f2e9f59f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:28.884990 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:28.886309 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:28.919435 29421 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:28.928714 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:28.929651 29155 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:28.938966 26538 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:15:28.992364 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:29.030506 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:29.211208 26424 consensus_queue.cc:237] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:29.213348 26555 raft_consensus.cc:1261] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:29.213475 29175 raft_consensus.cc:1261] T 9f3cbedfa1da42698615a544d9dc3276 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:29.213932 1965 consensus_queue.cc:1035] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:29.214231 2075 consensus_queue.cc:1035] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:29.217909 1965 raft_consensus.cc:2932] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:29.219120 29175 raft_consensus.cc:2932] T 9f3cbedfa1da42698615a544d9dc3276 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:29.227474 26283 catalog_manager.cc:5484] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: config changed from index 4 to 5, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:29.229261 26555 raft_consensus.cc:2932] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:29.246042 26669 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f3cbedfa1da42698615a544d9dc3276 with delete_type TABLET_DATA_TOMBSTONED (TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34150
I20240502 14:15:29.247150 2104 tablet_replica.cc:330] T 9f3cbedfa1da42698615a544d9dc3276 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: stopping tablet replica
I20240502 14:15:29.248812 2104 raft_consensus.cc:2229] T 9f3cbedfa1da42698615a544d9dc3276 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:29.249217 2104 raft_consensus.cc:2258] T 9f3cbedfa1da42698615a544d9dc3276 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:29.250139 2104 ts_tablet_manager.cc:1875] T 9f3cbedfa1da42698615a544d9dc3276 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:29.264180 2104 ts_tablet_manager.cc:1888] T 9f3cbedfa1da42698615a544d9dc3276 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:15:29.264508 2104 log.cc:1196] T 9f3cbedfa1da42698615a544d9dc3276 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/wals/9f3cbedfa1da42698615a544d9dc3276
I20240502 14:15:29.265846 26272 catalog_manager.cc:4830] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): tablet 9f3cbedfa1da42698615a544d9dc3276 (table rebalance_test_table_7 [id=998b08830b5f4b6fbb49d494f21739b9]) successfully deleted
I20240502 14:15:29.575238 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:29.581996 29421 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:29.585953 29155 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:29.640753 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:29.671010 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:29.671604 26538 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:15:29.707054 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:29.768720 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:29.889643 26426 consensus_queue.cc:237] T 2e682a5c24a64fe3ab7d325a027242d9 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:29.892738 29574 raft_consensus.cc:1261] T 2e682a5c24a64fe3ab7d325a027242d9 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:29.892925 26689 raft_consensus.cc:1261] T 2e682a5c24a64fe3ab7d325a027242d9 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:29.893482 2159 consensus_queue.cc:1035] T 2e682a5c24a64fe3ab7d325a027242d9 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:29.893821 1965 consensus_queue.cc:1035] T 2e682a5c24a64fe3ab7d325a027242d9 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:29.896783 2159 raft_consensus.cc:2932] T 2e682a5c24a64fe3ab7d325a027242d9 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:15:29.897429 26689 raft_consensus.cc:2932] T 2e682a5c24a64fe3ab7d325a027242d9 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:15:29.904433 26283 catalog_manager.cc:5484] T 2e682a5c24a64fe3ab7d325a027242d9 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:15:29.928459 29575 raft_consensus.cc:2932] T 2e682a5c24a64fe3ab7d325a027242d9 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:15:29.931910 29287 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e682a5c24a64fe3ab7d325a027242d9 with delete_type TABLET_DATA_TOMBSTONED (TS c1100e16b3b9495ca61d4f95e89b6a89 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:59160
I20240502 14:15:29.934581 2187 tablet_replica.cc:330] T 2e682a5c24a64fe3ab7d325a027242d9 P c1100e16b3b9495ca61d4f95e89b6a89: stopping tablet replica
I20240502 14:15:29.936581 2187 raft_consensus.cc:2229] T 2e682a5c24a64fe3ab7d325a027242d9 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:29.936964 2187 raft_consensus.cc:2258] T 2e682a5c24a64fe3ab7d325a027242d9 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:29.937780 2187 ts_tablet_manager.cc:1875] T 2e682a5c24a64fe3ab7d325a027242d9 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:29.944860 2187 ts_tablet_manager.cc:1888] T 2e682a5c24a64fe3ab7d325a027242d9 P c1100e16b3b9495ca61d4f95e89b6a89: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:15:29.945118 2187 log.cc:1196] T 2e682a5c24a64fe3ab7d325a027242d9 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal/wals/2e682a5c24a64fe3ab7d325a027242d9
I20240502 14:15:29.946369 26271 catalog_manager.cc:4830] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): tablet 2e682a5c24a64fe3ab7d325a027242d9 (table rebalancer_extra_table_32 [id=eb9f13fa5bdc4cc3b3c074bc196dfb8c]) successfully deleted
I20240502 14:15:30.314836 29155 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:30.351619 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:30.360807 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:30.365154 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:30.368649 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:30.409710 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:30.418292 26538 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:15:30.447137 29421 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:30.629870 26817 consensus_queue.cc:237] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:30.632360 26426 raft_consensus.cc:1261] T a42e68ab03f84d4b92893006fcf6f229 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:30.632488 26558 raft_consensus.cc:1261] T a42e68ab03f84d4b92893006fcf6f229 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:30.633021 1384 consensus_queue.cc:1035] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:30.648344 1384 consensus_queue.cc:1035] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:30.649745 1384 raft_consensus.cc:2932] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:30.650189 26558 raft_consensus.cc:2932] T a42e68ab03f84d4b92893006fcf6f229 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:30.657253 26278 catalog_manager.cc:5484] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: config changed from index 2 to 3, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) evicted. New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:30.677886 26424 raft_consensus.cc:2932] T a42e68ab03f84d4b92893006fcf6f229 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:30.687969 29155 tablet_service.cc:1514] Processing DeleteTablet for tablet a42e68ab03f84d4b92893006fcf6f229 with delete_type TABLET_DATA_TOMBSTONED (TS 1cb67158133e4648a6eceba32d4dbba3 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:55202
I20240502 14:15:30.688704 2271 tablet_replica.cc:330] T a42e68ab03f84d4b92893006fcf6f229 P 1cb67158133e4648a6eceba32d4dbba3: stopping tablet replica
I20240502 14:15:30.688977 2271 raft_consensus.cc:2229] T a42e68ab03f84d4b92893006fcf6f229 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:30.689285 2271 raft_consensus.cc:2258] T a42e68ab03f84d4b92893006fcf6f229 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:30.690418 2271 ts_tablet_manager.cc:1875] T a42e68ab03f84d4b92893006fcf6f229 P 1cb67158133e4648a6eceba32d4dbba3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:30.696154 2271 ts_tablet_manager.cc:1888] T a42e68ab03f84d4b92893006fcf6f229 P 1cb67158133e4648a6eceba32d4dbba3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:15:30.696298 2271 log.cc:1196] T a42e68ab03f84d4b92893006fcf6f229 P 1cb67158133e4648a6eceba32d4dbba3: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal/wals/a42e68ab03f84d4b92893006fcf6f229
I20240502 14:15:30.696950 26271 catalog_manager.cc:4830] TS 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): tablet a42e68ab03f84d4b92893006fcf6f229 (table rebalancer_extra_table_54 [id=36399b8bbc064edb98ef3de360e0e9bf]) successfully deleted
I20240502 14:15:30.983996 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:31.004464 29155 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:31.005764 29421 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:31.007113 26538 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:15:31.016440 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:31.057865 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:31.063355 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:31.078414 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:31.277416 29574 consensus_queue.cc:237] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:31.279878 29440 raft_consensus.cc:1261] T 9671eeb3b0e44730a4b0c2849e6ee012 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:31.279953 26818 raft_consensus.cc:1261] T 9671eeb3b0e44730a4b0c2849e6ee012 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:31.280802 2067 consensus_queue.cc:1035] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:31.281390 2067 consensus_queue.cc:1035] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:31.283874 2067 raft_consensus.cc:2932] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:31.284819 26818 raft_consensus.cc:2932] T 9671eeb3b0e44730a4b0c2849e6ee012 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:31.284927 29440 raft_consensus.cc:2932] T 9671eeb3b0e44730a4b0c2849e6ee012 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:31.292615 26277 catalog_manager.cc:5484] T 9671eeb3b0e44730a4b0c2849e6ee012 P 68a3c2b17ed94e06aba66ed4178132ab reported cstate change: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:31.314795 29287 tablet_service.cc:1514] Processing DeleteTablet for tablet 9671eeb3b0e44730a4b0c2849e6ee012 with delete_type TABLET_DATA_TOMBSTONED (TS c1100e16b3b9495ca61d4f95e89b6a89 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:59160
I20240502 14:15:31.330034 2353 tablet_replica.cc:330] T 9671eeb3b0e44730a4b0c2849e6ee012 P c1100e16b3b9495ca61d4f95e89b6a89: stopping tablet replica
I20240502 14:15:31.330298 2353 raft_consensus.cc:2229] T 9671eeb3b0e44730a4b0c2849e6ee012 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:31.330535 2353 raft_consensus.cc:2258] T 9671eeb3b0e44730a4b0c2849e6ee012 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:31.331105 2353 ts_tablet_manager.cc:1875] T 9671eeb3b0e44730a4b0c2849e6ee012 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:31.337064 2353 ts_tablet_manager.cc:1888] T 9671eeb3b0e44730a4b0c2849e6ee012 P c1100e16b3b9495ca61d4f95e89b6a89: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:15:31.337237 2353 log.cc:1196] T 9671eeb3b0e44730a4b0c2849e6ee012 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal/wals/9671eeb3b0e44730a4b0c2849e6ee012
I20240502 14:15:31.338443 26271 catalog_manager.cc:4830] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): tablet 9671eeb3b0e44730a4b0c2849e6ee012 (table rebalancer_extra_table_22 [id=10bc76506bf848969a3acfd452e07b7f]) successfully deleted
I20240502 14:15:31.647011 29155 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:31.670759 29421 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:31.671221 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 14:15:31.768055 29511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.199:41781, user_credentials={real_user=slave}} blocked reactor thread for 50430us
W20240502 14:15:31.838524 26755 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.200:46097, user_credentials={real_user=slave}} blocked reactor thread for 180821us
I20240502 14:15:31.895517 26538 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:15:31.899194 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:31.940981 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:31.967464 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:31.974331 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:32.129904 26555 consensus_queue.cc:237] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:32.132637 29574 raft_consensus.cc:1261] T 04f3eff43b544f33b020bd74e4331dea P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:32.132997 26689 raft_consensus.cc:1261] T 04f3eff43b544f33b020bd74e4331dea P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:32.133348 1359 consensus_queue.cc:1035] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:32.133863 1707 consensus_queue.cc:1035] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:32.149485 1707 raft_consensus.cc:2932] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:32.153463 26689 raft_consensus.cc:2932] T 04f3eff43b544f33b020bd74e4331dea P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:32.157047 26278 catalog_manager.cc:5484] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a reported cstate change: config changed from index 2 to 3, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:32.158685 29574 raft_consensus.cc:2932] T 04f3eff43b544f33b020bd74e4331dea P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:32.203056 26798 tablet_service.cc:1514] Processing DeleteTablet for tablet 04f3eff43b544f33b020bd74e4331dea with delete_type TABLET_DATA_TOMBSTONED (TS e4cfb8fc5a474177af6848a6efa13a27 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:56010
I20240502 14:15:32.213573 2451 tablet_replica.cc:330] T 04f3eff43b544f33b020bd74e4331dea P e4cfb8fc5a474177af6848a6efa13a27: stopping tablet replica
I20240502 14:15:32.213897 2451 raft_consensus.cc:2229] T 04f3eff43b544f33b020bd74e4331dea P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:32.214181 2451 raft_consensus.cc:2258] T 04f3eff43b544f33b020bd74e4331dea P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:32.215013 2451 ts_tablet_manager.cc:1875] T 04f3eff43b544f33b020bd74e4331dea P e4cfb8fc5a474177af6848a6efa13a27: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:32.223227 2451 ts_tablet_manager.cc:1888] T 04f3eff43b544f33b020bd74e4331dea P e4cfb8fc5a474177af6848a6efa13a27: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:15:32.223502 2451 log.cc:1196] T 04f3eff43b544f33b020bd74e4331dea P e4cfb8fc5a474177af6848a6efa13a27: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/wals/04f3eff43b544f33b020bd74e4331dea
I20240502 14:15:32.224483 26272 catalog_manager.cc:4830] TS e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): tablet 04f3eff43b544f33b020bd74e4331dea (table rebalancer_extra_table_50 [id=183571cb1d8d40aaa35266ced8ac1c76]) successfully deleted
I20240502 14:15:32.595649 29155 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:32.628938 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:32.672197 29421 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:32.768060 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:32.770541 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:32.810429 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:32.843294 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:32.850267 26538 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:15:33.016898 29440 consensus_queue.cc:237] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:33.029484 26557 raft_consensus.cc:1261] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:33.030426 29307 raft_consensus.cc:1261] T b8430c13474b4850b71020c4a89bea68 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:33.031222 2503 consensus_queue.cc:1035] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:33.031950 2503 consensus_queue.cc:1035] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:33.041643 2496 raft_consensus.cc:2932] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:15:33.047930 29309 raft_consensus.cc:2932] T b8430c13474b4850b71020c4a89bea68 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:15:33.051962 26277 catalog_manager.cc:5484] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab reported cstate change: config changed from index 2 to 3, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) evicted. New cstate: current_term: 1 leader_uuid: "68a3c2b17ed94e06aba66ed4178132ab" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:15:33.083880 26557 raft_consensus.cc:2932] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:15:33.098075 29554 tablet_service.cc:1514] Processing DeleteTablet for tablet b8430c13474b4850b71020c4a89bea68 with delete_type TABLET_DATA_TOMBSTONED (TS 4092902b3e674857a0147f1b4f2cdb86 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:36398
I20240502 14:15:33.098796 2540 tablet_replica.cc:330] T b8430c13474b4850b71020c4a89bea68 P 4092902b3e674857a0147f1b4f2cdb86: stopping tablet replica
I20240502 14:15:33.099081 2540 raft_consensus.cc:2229] T b8430c13474b4850b71020c4a89bea68 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:33.099375 2540 raft_consensus.cc:2258] T b8430c13474b4850b71020c4a89bea68 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:33.100433 2540 ts_tablet_manager.cc:1875] T b8430c13474b4850b71020c4a89bea68 P 4092902b3e674857a0147f1b4f2cdb86: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:33.114532 2540 ts_tablet_manager.cc:1888] T b8430c13474b4850b71020c4a89bea68 P 4092902b3e674857a0147f1b4f2cdb86: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:15:33.114728 2540 log.cc:1196] T b8430c13474b4850b71020c4a89bea68 P 4092902b3e674857a0147f1b4f2cdb86: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/wal/wals/b8430c13474b4850b71020c4a89bea68
I20240502 14:15:33.116457 26272 catalog_manager.cc:4830] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097): tablet b8430c13474b4850b71020c4a89bea68 (table rebalancer_extra_table_28 [id=0dc6f24955544368ae080f1a95f9589b]) successfully deleted
I20240502 14:15:33.622929 29155 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:33.641302 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:33.687470 29421 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:33.701974 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:33.704808 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:33.758291 26538 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:15:33.828058 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:33.848524 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:33.945926 29175 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f94a6ca24bf748c1ac4f12df761d537d"
dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52508
I20240502 14:15:33.946233 29175 raft_consensus.cc:595] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:15:33.971539 1604 raft_consensus.cc:979] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3: : Instructing follower ba187e38a2754d60a12de9ac4856430a to start an election
I20240502 14:15:33.971716 1604 raft_consensus.cc:1067] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3 [term 1 LEADER]: Signalling peer ba187e38a2754d60a12de9ac4856430a to start an election
I20240502 14:15:33.972895 26557 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f94a6ca24bf748c1ac4f12df761d537d"
dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
from {username='slave'} at 127.19.94.197:58769
I20240502 14:15:33.973135 26557 raft_consensus.cc:482] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:33.973237 26557 raft_consensus.cc:3037] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:33.975211 26557 raft_consensus.cc:504] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:33.976722 29175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f94a6ca24bf748c1ac4f12df761d537d" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:15:33.976773 29439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f94a6ca24bf748c1ac4f12df761d537d" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
I20240502 14:15:33.976956 29439 raft_consensus.cc:3037] T f94a6ca24bf748c1ac4f12df761d537d P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:33.976948 29175 raft_consensus.cc:3032] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:33.977026 29175 raft_consensus.cc:729] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Running, Role: LEADER
I20240502 14:15:33.977341 29175 consensus_queue.cc:260] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:33.977802 29175 raft_consensus.cc:3037] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:33.979714 29439 raft_consensus.cc:2454] T f94a6ca24bf748c1ac4f12df761d537d P 68a3c2b17ed94e06aba66ed4178132ab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 2.
I20240502 14:15:33.980453 29175 raft_consensus.cc:2454] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 2.
I20240502 14:15:33.980859 29307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f94a6ca24bf748c1ac4f12df761d537d" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
I20240502 14:15:33.981056 29307 raft_consensus.cc:3037] T f94a6ca24bf748c1ac4f12df761d537d P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:33.983517 29307 raft_consensus.cc:2454] T f94a6ca24bf748c1ac4f12df761d537d P c1100e16b3b9495ca61d4f95e89b6a89 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 2.
I20240502 14:15:33.983965 26493 leader_election.cc:302] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, ba187e38a2754d60a12de9ac4856430a, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:15:33.984299 1707 raft_consensus.cc:2781] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:33.984460 1707 raft_consensus.cc:686] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a [term 2 LEADER]: Becoming Leader. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Running, Role: LEADER
I20240502 14:15:33.984911 26557 leader_election.cc:288] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 election: Requested vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:15:33.984728 1707 consensus_queue.cc:237] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:33.989744 26284 catalog_manager.cc:5484] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a reported cstate change: term changed from 1 to 2, leader changed from 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) to ba187e38a2754d60a12de9ac4856430a (127.19.94.194). New cstate: current_term: 2 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:34.365887 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:34.418174 29175 raft_consensus.cc:1261] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:15:34.419723 1707 consensus_queue.cc:1035] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:34.485863 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:34.524150 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:34.537221 29439 raft_consensus.cc:1261] T f94a6ca24bf748c1ac4f12df761d537d P 68a3c2b17ed94e06aba66ed4178132ab [term 2 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:15:34.537948 1359 consensus_queue.cc:1035] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:34.564962 29421 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:34.584707 29307 raft_consensus.cc:1261] T f94a6ca24bf748c1ac4f12df761d537d P c1100e16b3b9495ca61d4f95e89b6a89 [term 2 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:15:34.585669 1359 consensus_queue.cc:1035] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:34.686118 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:34.687898 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:34.715709 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:34.722292 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:35.362951 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:35.389233 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:35.478120 29421 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:35.516626 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:35.537643 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:35.574137 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:35.610433 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:35.648698 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:35.830468 29573 consensus_queue.cc:237] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:35.833077 26424 raft_consensus.cc:1261] T 0c7ccae60f58427fbd39c26b98817963 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:35.833103 29439 raft_consensus.cc:1261] T 0c7ccae60f58427fbd39c26b98817963 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:35.833684 2217 consensus_queue.cc:1035] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:35.834306 2772 consensus_queue.cc:1035] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:35.837306 2217 raft_consensus.cc:2932] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:35.837869 29439 raft_consensus.cc:2932] T 0c7ccae60f58427fbd39c26b98817963 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:35.841643 26424 raft_consensus.cc:2932] T 0c7ccae60f58427fbd39c26b98817963 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:35.846043 26281 catalog_manager.cc:5484] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: config changed from index 2 to 3, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:35.856307 26669 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c7ccae60f58427fbd39c26b98817963 with delete_type TABLET_DATA_TOMBSTONED (TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:34150
I20240502 14:15:35.857897 2804 tablet_replica.cc:330] T 0c7ccae60f58427fbd39c26b98817963 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: stopping tablet replica
I20240502 14:15:35.858186 2804 raft_consensus.cc:2229] T 0c7ccae60f58427fbd39c26b98817963 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:35.858433 2804 raft_consensus.cc:2258] T 0c7ccae60f58427fbd39c26b98817963 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:35.859263 2804 ts_tablet_manager.cc:1875] T 0c7ccae60f58427fbd39c26b98817963 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:35.867071 2804 ts_tablet_manager.cc:1888] T 0c7ccae60f58427fbd39c26b98817963 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:15:35.867281 2804 log.cc:1196] T 0c7ccae60f58427fbd39c26b98817963 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/wals/0c7ccae60f58427fbd39c26b98817963
I20240502 14:15:35.868732 26272 catalog_manager.cc:4830] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): tablet 0c7ccae60f58427fbd39c26b98817963 (table rebalancer_extra_table_2 [id=df45ec00aaba41b2b3f62c6fb9633ae0]) successfully deleted
I20240502 14:15:36.281569 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:36.299554 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:36.301373 29421 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:36.315196 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:36.323885 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:36.331183 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:36.361893 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:36.444684 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:36.611828 26689 consensus_queue.cc:237] T 0f2a4042e0f64ceebb345b9cf9764805 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:36.614259 26557 raft_consensus.cc:1261] T 0f2a4042e0f64ceebb345b9cf9764805 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:36.614274 26818 raft_consensus.cc:1261] T 0f2a4042e0f64ceebb345b9cf9764805 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:36.614918 2652 consensus_queue.cc:1035] T 0f2a4042e0f64ceebb345b9cf9764805 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:36.615278 1022 consensus_queue.cc:1035] T 0f2a4042e0f64ceebb345b9cf9764805 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:36.621227 1022 raft_consensus.cc:2932] T 0f2a4042e0f64ceebb345b9cf9764805 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:36.621955 26818 raft_consensus.cc:2932] T 0f2a4042e0f64ceebb345b9cf9764805 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:36.635545 26284 catalog_manager.cc:5484] T 0f2a4042e0f64ceebb345b9cf9764805 P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: config changed from index 2 to 3, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) evicted. New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:36.668018 26557 raft_consensus.cc:2932] T 0f2a4042e0f64ceebb345b9cf9764805 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:36.669688 26406 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f2a4042e0f64ceebb345b9cf9764805 with delete_type TABLET_DATA_TOMBSTONED (TS 89af7f423cf44717a08d6b142f0ed49d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:37422
I20240502 14:15:36.670503 2890 tablet_replica.cc:330] T 0f2a4042e0f64ceebb345b9cf9764805 P 89af7f423cf44717a08d6b142f0ed49d: stopping tablet replica
I20240502 14:15:36.670883 2890 raft_consensus.cc:2229] T 0f2a4042e0f64ceebb345b9cf9764805 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:36.671564 2890 raft_consensus.cc:2258] T 0f2a4042e0f64ceebb345b9cf9764805 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:36.672547 2890 ts_tablet_manager.cc:1875] T 0f2a4042e0f64ceebb345b9cf9764805 P 89af7f423cf44717a08d6b142f0ed49d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:36.679718 2890 ts_tablet_manager.cc:1888] T 0f2a4042e0f64ceebb345b9cf9764805 P 89af7f423cf44717a08d6b142f0ed49d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:15:36.679929 2890 log.cc:1196] T 0f2a4042e0f64ceebb345b9cf9764805 P 89af7f423cf44717a08d6b142f0ed49d: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/0f2a4042e0f64ceebb345b9cf9764805
I20240502 14:15:36.680754 26271 catalog_manager.cc:4830] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): tablet 0f2a4042e0f64ceebb345b9cf9764805 (table rebalancer_extra_table_48 [id=53d47c43f2f14c30b29a5147461aafae]) successfully deleted
I20240502 14:15:36.976626 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:37.006188 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:37.022075 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:37.046732 29421 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:37.067518 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240502 14:15:37.099453 29245 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.199:41781, user_credentials={real_user=slave}} blocked reactor thread for 86862.2us
I20240502 14:15:37.128890 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:37.298871 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:37.325007 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:37.464538 29306 consensus_queue.cc:237] T abdd0f01d8d5453793c7e9461af10516 P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:37.468190 29439 raft_consensus.cc:1261] T abdd0f01d8d5453793c7e9461af10516 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer c1100e16b3b9495ca61d4f95e89b6a89: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:37.469135 2074 consensus_queue.cc:1035] T abdd0f01d8d5453793c7e9461af10516 P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:37.470381 26555 raft_consensus.cc:1261] T abdd0f01d8d5453793c7e9461af10516 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer c1100e16b3b9495ca61d4f95e89b6a89: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:37.471042 2950 consensus_queue.cc:1035] T abdd0f01d8d5453793c7e9461af10516 P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:37.474231 2954 raft_consensus.cc:2932] T abdd0f01d8d5453793c7e9461af10516 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:37.477543 26555 raft_consensus.cc:2932] T abdd0f01d8d5453793c7e9461af10516 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:37.483229 29439 raft_consensus.cc:2932] T abdd0f01d8d5453793c7e9461af10516 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:37.485325 26277 catalog_manager.cc:5484] T abdd0f01d8d5453793c7e9461af10516 P c1100e16b3b9495ca61d4f95e89b6a89 reported cstate change: config changed from index 2 to 3, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) evicted. New cstate: current_term: 1 leader_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:37.513181 29554 tablet_service.cc:1514] Processing DeleteTablet for tablet abdd0f01d8d5453793c7e9461af10516 with delete_type TABLET_DATA_TOMBSTONED (TS 4092902b3e674857a0147f1b4f2cdb86 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:36398
I20240502 14:15:37.513851 2980 tablet_replica.cc:330] T abdd0f01d8d5453793c7e9461af10516 P 4092902b3e674857a0147f1b4f2cdb86: stopping tablet replica
I20240502 14:15:37.514060 2980 raft_consensus.cc:2229] T abdd0f01d8d5453793c7e9461af10516 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:37.514293 2980 raft_consensus.cc:2258] T abdd0f01d8d5453793c7e9461af10516 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:37.514874 2980 ts_tablet_manager.cc:1875] T abdd0f01d8d5453793c7e9461af10516 P 4092902b3e674857a0147f1b4f2cdb86: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:37.519795 2980 ts_tablet_manager.cc:1888] T abdd0f01d8d5453793c7e9461af10516 P 4092902b3e674857a0147f1b4f2cdb86: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:15:37.519939 2980 log.cc:1196] T abdd0f01d8d5453793c7e9461af10516 P 4092902b3e674857a0147f1b4f2cdb86: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/wal/wals/abdd0f01d8d5453793c7e9461af10516
I20240502 14:15:37.520869 26272 catalog_manager.cc:4830] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097): tablet abdd0f01d8d5453793c7e9461af10516 (table rebalancer_extra_table_2 [id=df45ec00aaba41b2b3f62c6fb9633ae0]) successfully deleted
I20240502 14:15:37.817644 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:37.855443 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:37.879565 29421 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:37.907430 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:37.908834 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:37.923945 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:37.924255 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:37.957305 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:38.186321 26815 consensus_queue.cc:237] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:38.189006 29175 raft_consensus.cc:1261] T 39a61744642f40c8ae92da708dc91ca3 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:38.189018 26423 raft_consensus.cc:1261] T 39a61744642f40c8ae92da708dc91ca3 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:38.189698 2546 consensus_queue.cc:1035] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:38.190228 2546 consensus_queue.cc:1035] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:38.193274 2750 raft_consensus.cc:2932] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:38.193774 29176 raft_consensus.cc:2932] T 39a61744642f40c8ae92da708dc91ca3 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:38.201897 26285 catalog_manager.cc:5484] T 39a61744642f40c8ae92da708dc91ca3 P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: config changed from index 4 to 5, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:38.204511 26423 raft_consensus.cc:2932] T 39a61744642f40c8ae92da708dc91ca3 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:38.235183 26669 tablet_service.cc:1514] Processing DeleteTablet for tablet 39a61744642f40c8ae92da708dc91ca3 with delete_type TABLET_DATA_TOMBSTONED (TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34150
I20240502 14:15:38.235966 3066 tablet_replica.cc:330] T 39a61744642f40c8ae92da708dc91ca3 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: stopping tablet replica
I20240502 14:15:38.236554 3066 raft_consensus.cc:2229] T 39a61744642f40c8ae92da708dc91ca3 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:38.236922 3066 raft_consensus.cc:2258] T 39a61744642f40c8ae92da708dc91ca3 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:38.237919 3066 ts_tablet_manager.cc:1875] T 39a61744642f40c8ae92da708dc91ca3 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:38.245072 3066 ts_tablet_manager.cc:1888] T 39a61744642f40c8ae92da708dc91ca3 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:15:38.245334 3066 log.cc:1196] T 39a61744642f40c8ae92da708dc91ca3 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/wals/39a61744642f40c8ae92da708dc91ca3
I20240502 14:15:38.246409 26272 catalog_manager.cc:4830] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): tablet 39a61744642f40c8ae92da708dc91ca3 (table rebalance_test_table_3 [id=e814fcdd8ac94d6eb54d1c57f004c870]) successfully deleted
I20240502 14:15:38.603797 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:38.655076 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:38.663947 29421 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:38.692744 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:38.721024 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:38.722914 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:38.750873 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:38.790715 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:38.932178 29306 consensus_queue.cc:237] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:38.937772 26423 raft_consensus.cc:1261] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer c1100e16b3b9495ca61d4f95e89b6a89: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:38.937772 26555 raft_consensus.cc:1261] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer c1100e16b3b9495ca61d4f95e89b6a89: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:38.938446 3131 consensus_queue.cc:1035] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:38.938819 2954 consensus_queue.cc:1035] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:38.944001 3037 raft_consensus.cc:2932] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:38.944857 26423 raft_consensus.cc:2932] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:38.944981 26558 raft_consensus.cc:2932] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:38.950767 26280 catalog_manager.cc:5484] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P c1100e16b3b9495ca61d4f95e89b6a89 reported cstate change: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New cstate: current_term: 1 leader_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:38.966370 29421 tablet_service.cc:1514] Processing DeleteTablet for tablet 842d4e7e5ee74e309fb7eba6d7b7ae8b with delete_type TABLET_DATA_TOMBSTONED (TS 68a3c2b17ed94e06aba66ed4178132ab not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:40652
I20240502 14:15:38.967121 3159 tablet_replica.cc:330] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 68a3c2b17ed94e06aba66ed4178132ab: stopping tablet replica
I20240502 14:15:38.967348 3159 raft_consensus.cc:2229] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:38.967578 3159 raft_consensus.cc:2258] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:38.968333 3159 ts_tablet_manager.cc:1875] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 68a3c2b17ed94e06aba66ed4178132ab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:38.974809 3159 ts_tablet_manager.cc:1888] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 68a3c2b17ed94e06aba66ed4178132ab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:15:38.975083 3159 log.cc:1196] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 68a3c2b17ed94e06aba66ed4178132ab: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal/wals/842d4e7e5ee74e309fb7eba6d7b7ae8b
I20240502 14:15:38.975960 26272 catalog_manager.cc:4830] TS 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): tablet 842d4e7e5ee74e309fb7eba6d7b7ae8b (table rebalancer_extra_table_52 [id=65a874546db5437ea59e74602d9071de]) successfully deleted
I20240502 14:15:39.384093 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:39.387710 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:39.433420 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:39.435237 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:39.481321 29421 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:39.482646 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:39.493754 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:39.522747 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:39.734810 29573 consensus_queue.cc:237] T 5808efbc3ef84f3b862045c67b30cd8b P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:39.739180 26816 raft_consensus.cc:1261] T 5808efbc3ef84f3b862045c67b30cd8b P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:39.739197 29175 raft_consensus.cc:1261] T 5808efbc3ef84f3b862045c67b30cd8b P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:39.739828 2937 consensus_queue.cc:1035] T 5808efbc3ef84f3b862045c67b30cd8b P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:39.740159 3228 consensus_queue.cc:1035] T 5808efbc3ef84f3b862045c67b30cd8b P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:39.747002 3228 raft_consensus.cc:2932] T 5808efbc3ef84f3b862045c67b30cd8b P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:39.750684 29175 raft_consensus.cc:2932] T 5808efbc3ef84f3b862045c67b30cd8b P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:39.751603 26816 raft_consensus.cc:2932] T 5808efbc3ef84f3b862045c67b30cd8b P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:39.757447 26281 catalog_manager.cc:5484] T 5808efbc3ef84f3b862045c67b30cd8b P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:39.776335 29287 tablet_service.cc:1514] Processing DeleteTablet for tablet 5808efbc3ef84f3b862045c67b30cd8b with delete_type TABLET_DATA_TOMBSTONED (TS c1100e16b3b9495ca61d4f95e89b6a89 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:59160
I20240502 14:15:39.777086 3254 tablet_replica.cc:330] T 5808efbc3ef84f3b862045c67b30cd8b P c1100e16b3b9495ca61d4f95e89b6a89: stopping tablet replica
I20240502 14:15:39.777482 3254 raft_consensus.cc:2229] T 5808efbc3ef84f3b862045c67b30cd8b P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:39.777715 3254 raft_consensus.cc:2258] T 5808efbc3ef84f3b862045c67b30cd8b P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:39.778512 3254 ts_tablet_manager.cc:1875] T 5808efbc3ef84f3b862045c67b30cd8b P c1100e16b3b9495ca61d4f95e89b6a89: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:39.786732 3254 ts_tablet_manager.cc:1888] T 5808efbc3ef84f3b862045c67b30cd8b P c1100e16b3b9495ca61d4f95e89b6a89: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:15:39.786899 3254 log.cc:1196] T 5808efbc3ef84f3b862045c67b30cd8b P c1100e16b3b9495ca61d4f95e89b6a89: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal/wals/5808efbc3ef84f3b862045c67b30cd8b
I20240502 14:15:39.787811 26271 catalog_manager.cc:4830] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): tablet 5808efbc3ef84f3b862045c67b30cd8b (table rebalancer_extra_table_30 [id=50fafe33ec534b21a087e7ea3eab5a91]) successfully deleted
I20240502 14:15:40.091910 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:40.101912 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:40.180960 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:40.222728 29421 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:40.282541 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:40.300158 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:40.309520 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:40.358944 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:40.534189 26815 consensus_queue.cc:237] T 0d0298cfb57342f7b840a8656f452645 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:40.536974 26427 raft_consensus.cc:1261] T 0d0298cfb57342f7b840a8656f452645 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:40.537106 29574 raft_consensus.cc:1261] T 0d0298cfb57342f7b840a8656f452645 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:40.537532 3057 consensus_queue.cc:1035] T 0d0298cfb57342f7b840a8656f452645 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:40.537837 3315 consensus_queue.cc:1035] T 0d0298cfb57342f7b840a8656f452645 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:40.540838 2750 raft_consensus.cc:2932] T 0d0298cfb57342f7b840a8656f452645 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:40.541961 29574 raft_consensus.cc:2932] T 0d0298cfb57342f7b840a8656f452645 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:40.548583 26427 raft_consensus.cc:2932] T 0d0298cfb57342f7b840a8656f452645 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:40.549170 26281 catalog_manager.cc:5484] T 0d0298cfb57342f7b840a8656f452645 P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:40.571424 29421 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d0298cfb57342f7b840a8656f452645 with delete_type TABLET_DATA_TOMBSTONED (TS 68a3c2b17ed94e06aba66ed4178132ab not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:40652
I20240502 14:15:40.572118 3343 tablet_replica.cc:330] T 0d0298cfb57342f7b840a8656f452645 P 68a3c2b17ed94e06aba66ed4178132ab: stopping tablet replica
I20240502 14:15:40.572422 3343 raft_consensus.cc:2229] T 0d0298cfb57342f7b840a8656f452645 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:40.572691 3343 raft_consensus.cc:2258] T 0d0298cfb57342f7b840a8656f452645 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:40.573421 3343 ts_tablet_manager.cc:1875] T 0d0298cfb57342f7b840a8656f452645 P 68a3c2b17ed94e06aba66ed4178132ab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:40.580910 3343 ts_tablet_manager.cc:1888] T 0d0298cfb57342f7b840a8656f452645 P 68a3c2b17ed94e06aba66ed4178132ab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:15:40.581063 3343 log.cc:1196] T 0d0298cfb57342f7b840a8656f452645 P 68a3c2b17ed94e06aba66ed4178132ab: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal/wals/0d0298cfb57342f7b840a8656f452645
I20240502 14:15:40.581878 26272 catalog_manager.cc:4830] TS 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): tablet 0d0298cfb57342f7b840a8656f452645 (table rebalancer_extra_table_14 [id=312a6520544b4149a5813f3f7652f51f]) successfully deleted
I20240502 14:15:40.973235 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:41.004964 29421 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:41.007438 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:41.020947 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:41.023146 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:41.132752 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:41.189713 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:41.191277 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:41.338086 26687 consensus_queue.cc:237] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:41.341149 29440 raft_consensus.cc:1261] T cf52d0f868fa45418af48639e12f3282 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:41.341320 26555 raft_consensus.cc:1261] T cf52d0f868fa45418af48639e12f3282 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:41.341836 2883 consensus_queue.cc:1035] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:41.342196 2652 consensus_queue.cc:1035] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:41.362208 2652 raft_consensus.cc:2932] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:15:41.369798 26558 raft_consensus.cc:2932] T cf52d0f868fa45418af48639e12f3282 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:15:41.375958 26283 catalog_manager.cc:5484] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: config changed from index 4 to 5, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:15:41.381294 29440 raft_consensus.cc:2932] T cf52d0f868fa45418af48639e12f3282 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:15:41.417728 26798 tablet_service.cc:1514] Processing DeleteTablet for tablet cf52d0f868fa45418af48639e12f3282 with delete_type TABLET_DATA_TOMBSTONED (TS e4cfb8fc5a474177af6848a6efa13a27 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:56010
I20240502 14:15:41.418406 3450 tablet_replica.cc:330] T cf52d0f868fa45418af48639e12f3282 P e4cfb8fc5a474177af6848a6efa13a27: stopping tablet replica
I20240502 14:15:41.418617 3450 raft_consensus.cc:2229] T cf52d0f868fa45418af48639e12f3282 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:41.418893 3450 raft_consensus.cc:2258] T cf52d0f868fa45418af48639e12f3282 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:41.420754 3450 ts_tablet_manager.cc:1875] T cf52d0f868fa45418af48639e12f3282 P e4cfb8fc5a474177af6848a6efa13a27: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:41.427582 3450 ts_tablet_manager.cc:1888] T cf52d0f868fa45418af48639e12f3282 P e4cfb8fc5a474177af6848a6efa13a27: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:15:41.427947 3450 log.cc:1196] T cf52d0f868fa45418af48639e12f3282 P e4cfb8fc5a474177af6848a6efa13a27: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/wals/cf52d0f868fa45418af48639e12f3282
I20240502 14:15:41.432080 26272 catalog_manager.cc:4830] TS e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): tablet cf52d0f868fa45418af48639e12f3282 (table rebalance_test_table_19 [id=d4a5cfed144a42db8322f6823ef341a8]) successfully deleted
I20240502 14:15:41.750828 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:41.782558 29421 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:41.799274 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:41.800581 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:41.803815 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:41.876677 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:41.926539 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:41.954334 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:42.116236 29175 consensus_queue.cc:237] T 39e6a71b3eef40848ea1ba4ef9604fa2 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:42.118820 26815 raft_consensus.cc:1261] T 39e6a71b3eef40848ea1ba4ef9604fa2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:42.118824 26555 raft_consensus.cc:1261] T 39e6a71b3eef40848ea1ba4ef9604fa2 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:42.119467 3519 consensus_queue.cc:1035] T 39e6a71b3eef40848ea1ba4ef9604fa2 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:42.119786 2956 consensus_queue.cc:1035] T 39e6a71b3eef40848ea1ba4ef9604fa2 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:42.122900 3519 raft_consensus.cc:2932] T 39e6a71b3eef40848ea1ba4ef9604fa2 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:42.123626 26555 raft_consensus.cc:2932] T 39e6a71b3eef40848ea1ba4ef9604fa2 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:42.127431 26815 raft_consensus.cc:2932] T 39e6a71b3eef40848ea1ba4ef9604fa2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:42.133888 26283 catalog_manager.cc:5484] T 39e6a71b3eef40848ea1ba4ef9604fa2 P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New cstate: current_term: 1 leader_uuid: "1cb67158133e4648a6eceba32d4dbba3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:42.147413 29287 tablet_service.cc:1514] Processing DeleteTablet for tablet 39e6a71b3eef40848ea1ba4ef9604fa2 with delete_type TABLET_DATA_TOMBSTONED (TS c1100e16b3b9495ca61d4f95e89b6a89 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:59160
I20240502 14:15:42.148983 3544 tablet_replica.cc:330] T 39e6a71b3eef40848ea1ba4ef9604fa2 P c1100e16b3b9495ca61d4f95e89b6a89: stopping tablet replica
I20240502 14:15:42.149226 3544 raft_consensus.cc:2229] T 39e6a71b3eef40848ea1ba4ef9604fa2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:42.149415 3544 raft_consensus.cc:2258] T 39e6a71b3eef40848ea1ba4ef9604fa2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:42.150007 3544 ts_tablet_manager.cc:1875] T 39e6a71b3eef40848ea1ba4ef9604fa2 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:42.168968 3544 ts_tablet_manager.cc:1888] T 39e6a71b3eef40848ea1ba4ef9604fa2 P c1100e16b3b9495ca61d4f95e89b6a89: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:15:42.169286 3544 log.cc:1196] T 39e6a71b3eef40848ea1ba4ef9604fa2 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal/wals/39e6a71b3eef40848ea1ba4ef9604fa2
I20240502 14:15:42.172159 26271 catalog_manager.cc:4830] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): tablet 39e6a71b3eef40848ea1ba4ef9604fa2 (table rebalancer_extra_table_4 [id=052077f1ff8540afbb74e3609ab56e61]) successfully deleted
I20240502 14:15:42.242383 29175 consensus_queue.cc:237] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:42.253131 26815 raft_consensus.cc:1261] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:15:42.253268 29112 consensus_peers.cc:485] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c32ea761f2834d758529890e5b496f88. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:42.253387 29306 raft_consensus.cc:1261] T c32ea761f2834d758529890e5b496f88 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:42.253647 3518 consensus_queue.cc:1035] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:42.253954 3519 consensus_queue.cc:1035] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:42.257530 3519 raft_consensus.cc:2932] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:15:42.258729 29306 raft_consensus.cc:2932] T c32ea761f2834d758529890e5b496f88 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:15:42.263617 26815 raft_consensus.cc:2932] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:15:42.266150 26284 catalog_manager.cc:5484] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: config changed from index -1 to 2, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New cstate: current_term: 1 leader_uuid: "1cb67158133e4648a6eceba32d4dbba3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:15:42.305186 26689 consensus_queue.cc:237] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:42.307832 29440 raft_consensus.cc:1261] T 870100caa8bf4ca1b49337ba374b1f83 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:15:42.308282 26624 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): Couldn't send request to peer ba187e38a2754d60a12de9ac4856430a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 870100caa8bf4ca1b49337ba374b1f83. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:42.308425 26426 raft_consensus.cc:1261] T 870100caa8bf4ca1b49337ba374b1f83 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:42.308920 3585 consensus_queue.cc:1035] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:42.309302 3441 consensus_queue.cc:1035] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:42.321861 3585 raft_consensus.cc:2932] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:42.322599 26426 raft_consensus.cc:2932] T 870100caa8bf4ca1b49337ba374b1f83 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:42.331069 26278 catalog_manager.cc:5484] T 870100caa8bf4ca1b49337ba374b1f83 P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:42.333050 29440 raft_consensus.cc:2932] T 870100caa8bf4ca1b49337ba374b1f83 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:42.688345 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:42.692799 3641 ts_tablet_manager.cc:902] T c32ea761f2834d758529890e5b496f88 P 68a3c2b17ed94e06aba66ed4178132ab: Initiating tablet copy from peer 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:15:42.722929 3641 tablet_copy_client.cc:308] T c32ea761f2834d758529890e5b496f88 P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.197:38715
I20240502 14:15:42.723800 29196 tablet_copy_service.cc:140] P 1cb67158133e4648a6eceba32d4dbba3: Received BeginTabletCopySession request for tablet c32ea761f2834d758529890e5b496f88 from peer 68a3c2b17ed94e06aba66ed4178132ab ({username='slave'} at 127.19.94.199:40161)
I20240502 14:15:42.724012 29196 tablet_copy_service.cc:161] P 1cb67158133e4648a6eceba32d4dbba3: Beginning new tablet copy session on tablet c32ea761f2834d758529890e5b496f88 from peer 68a3c2b17ed94e06aba66ed4178132ab at {username='slave'} at 127.19.94.199:40161: session id = 68a3c2b17ed94e06aba66ed4178132ab-c32ea761f2834d758529890e5b496f88
I20240502 14:15:42.727005 29196 tablet_copy_source_session.cc:205] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:42.728216 29421 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:42.730549 3641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c32ea761f2834d758529890e5b496f88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:42.755116 3641 tablet_copy_client.cc:783] T c32ea761f2834d758529890e5b496f88 P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:42.756271 3641 tablet_copy_client.cc:647] T c32ea761f2834d758529890e5b496f88 P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:42.761407 3641 tablet_copy_client.cc:523] T c32ea761f2834d758529890e5b496f88 P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:42.770465 3641 tablet_bootstrap.cc:492] T c32ea761f2834d758529890e5b496f88 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:15:42.774246 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:42.774425 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:42.848704 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:42.888322 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:42.907231 3641 tablet_bootstrap.cc:492] T c32ea761f2834d758529890e5b496f88 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:42.907694 3641 tablet_bootstrap.cc:492] T c32ea761f2834d758529890e5b496f88 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap complete.
I20240502 14:15:42.907958 3641 ts_tablet_manager.cc:1367] T c32ea761f2834d758529890e5b496f88 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.138s user 0.008s sys 0.005s
I20240502 14:15:42.913847 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 14:15:42.925114 26362 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.194:36025, user_credentials={real_user=slave}} blocked reactor thread for 98886.3us
W20240502 14:15:42.926352 26754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.197:38715, user_credentials={real_user=slave}} blocked reactor thread for 111695us
I20240502 14:15:42.930413 3641 raft_consensus.cc:348] T c32ea761f2834d758529890e5b496f88 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:42.930943 3641 raft_consensus.cc:729] T c32ea761f2834d758529890e5b496f88 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:15:42.931265 3641 consensus_queue.cc:260] T c32ea761f2834d758529890e5b496f88 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:42.932186 3657 ts_tablet_manager.cc:902] T 870100caa8bf4ca1b49337ba374b1f83 P ba187e38a2754d60a12de9ac4856430a: Initiating tablet copy from peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:15:42.933959 3641 ts_tablet_manager.cc:1398] T c32ea761f2834d758529890e5b496f88 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.026s user 0.005s sys 0.000s
I20240502 14:15:42.935956 29196 tablet_copy_service.cc:342] P 1cb67158133e4648a6eceba32d4dbba3: Request end of tablet copy session 68a3c2b17ed94e06aba66ed4178132ab-c32ea761f2834d758529890e5b496f88 received from {username='slave'} at 127.19.94.199:40161
I20240502 14:15:42.936156 29196 tablet_copy_service.cc:433] P 1cb67158133e4648a6eceba32d4dbba3: ending tablet copy session 68a3c2b17ed94e06aba66ed4178132ab-c32ea761f2834d758529890e5b496f88 on tablet c32ea761f2834d758529890e5b496f88 with peer 68a3c2b17ed94e06aba66ed4178132ab
I20240502 14:15:42.974223 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:43.010777 3657 tablet_copy_client.cc:308] T 870100caa8bf4ca1b49337ba374b1f83 P ba187e38a2754d60a12de9ac4856430a: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.195:44325
I20240502 14:15:43.011883 26709 tablet_copy_service.cc:140] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Received BeginTabletCopySession request for tablet 870100caa8bf4ca1b49337ba374b1f83 from peer ba187e38a2754d60a12de9ac4856430a ({username='slave'} at 127.19.94.194:58393)
I20240502 14:15:43.012068 26709 tablet_copy_service.cc:161] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Beginning new tablet copy session on tablet 870100caa8bf4ca1b49337ba374b1f83 from peer ba187e38a2754d60a12de9ac4856430a at {username='slave'} at 127.19.94.194:58393: session id = ba187e38a2754d60a12de9ac4856430a-870100caa8bf4ca1b49337ba374b1f83
I20240502 14:15:43.015113 26709 tablet_copy_source_session.cc:205] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:43.017649 3657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 870100caa8bf4ca1b49337ba374b1f83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:43.027290 3657 tablet_copy_client.cc:783] T 870100caa8bf4ca1b49337ba374b1f83 P ba187e38a2754d60a12de9ac4856430a: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:43.027889 3657 tablet_copy_client.cc:647] T 870100caa8bf4ca1b49337ba374b1f83 P ba187e38a2754d60a12de9ac4856430a: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:43.030404 3657 tablet_copy_client.cc:523] T 870100caa8bf4ca1b49337ba374b1f83 P ba187e38a2754d60a12de9ac4856430a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:43.037839 3657 tablet_bootstrap.cc:492] T 870100caa8bf4ca1b49337ba374b1f83 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:15:43.049145 3657 tablet_bootstrap.cc:492] T 870100caa8bf4ca1b49337ba374b1f83 P ba187e38a2754d60a12de9ac4856430a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:43.049856 3657 tablet_bootstrap.cc:492] T 870100caa8bf4ca1b49337ba374b1f83 P ba187e38a2754d60a12de9ac4856430a: Bootstrap complete.
I20240502 14:15:43.050369 3657 ts_tablet_manager.cc:1367] T 870100caa8bf4ca1b49337ba374b1f83 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.004s
I20240502 14:15:43.051198 3657 raft_consensus.cc:348] T 870100caa8bf4ca1b49337ba374b1f83 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:43.051512 3657 raft_consensus.cc:729] T 870100caa8bf4ca1b49337ba374b1f83 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:15:43.051841 3657 consensus_queue.cc:260] T 870100caa8bf4ca1b49337ba374b1f83 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:43.052628 3657 ts_tablet_manager.cc:1398] T 870100caa8bf4ca1b49337ba374b1f83 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:15:43.054214 26709 tablet_copy_service.cc:342] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Request end of tablet copy session ba187e38a2754d60a12de9ac4856430a-870100caa8bf4ca1b49337ba374b1f83 received from {username='slave'} at 127.19.94.194:58393
I20240502 14:15:43.054363 26709 tablet_copy_service.cc:433] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: ending tablet copy session ba187e38a2754d60a12de9ac4856430a-870100caa8bf4ca1b49337ba374b1f83 on tablet 870100caa8bf4ca1b49337ba374b1f83 with peer ba187e38a2754d60a12de9ac4856430a
I20240502 14:15:43.244717 29439 raft_consensus.cc:1203] T c32ea761f2834d758529890e5b496f88 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:43.391425 26555 raft_consensus.cc:1203] T 870100caa8bf4ca1b49337ba374b1f83 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:43.480983 29421 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:43.487357 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:43.490332 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:43.490574 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:43.519927 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:43.544955 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:43.608644 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:43.625257 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:43.794399 26816 consensus_queue.cc:237] T a823ce0b7b174b53902a85b7f2e9f59f P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:43.796700 29176 raft_consensus.cc:1261] T a823ce0b7b174b53902a85b7f2e9f59f P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:43.797019 26687 raft_consensus.cc:1261] T a823ce0b7b174b53902a85b7f2e9f59f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:43.797323 3732 consensus_queue.cc:1035] T a823ce0b7b174b53902a85b7f2e9f59f P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:43.797617 3733 consensus_queue.cc:1035] T a823ce0b7b174b53902a85b7f2e9f59f P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:43.800781 3732 raft_consensus.cc:2932] T a823ce0b7b174b53902a85b7f2e9f59f P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:43.801561 26687 raft_consensus.cc:2932] T a823ce0b7b174b53902a85b7f2e9f59f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:43.802168 29176 raft_consensus.cc:2932] T a823ce0b7b174b53902a85b7f2e9f59f P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:43.808224 26278 catalog_manager.cc:5484] T a823ce0b7b174b53902a85b7f2e9f59f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: config changed from index 2 to 3, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) evicted. New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:43.820873 29554 tablet_service.cc:1514] Processing DeleteTablet for tablet a823ce0b7b174b53902a85b7f2e9f59f with delete_type TABLET_DATA_TOMBSTONED (TS 4092902b3e674857a0147f1b4f2cdb86 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:36398
I20240502 14:15:43.821558 3758 tablet_replica.cc:330] T a823ce0b7b174b53902a85b7f2e9f59f P 4092902b3e674857a0147f1b4f2cdb86: stopping tablet replica
I20240502 14:15:43.821887 3758 raft_consensus.cc:2229] T a823ce0b7b174b53902a85b7f2e9f59f P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:43.822219 3758 raft_consensus.cc:2258] T a823ce0b7b174b53902a85b7f2e9f59f P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:43.823621 3758 ts_tablet_manager.cc:1875] T a823ce0b7b174b53902a85b7f2e9f59f P 4092902b3e674857a0147f1b4f2cdb86: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:43.830260 3758 ts_tablet_manager.cc:1888] T a823ce0b7b174b53902a85b7f2e9f59f P 4092902b3e674857a0147f1b4f2cdb86: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:15:43.831128 3758 log.cc:1196] T a823ce0b7b174b53902a85b7f2e9f59f P 4092902b3e674857a0147f1b4f2cdb86: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/wal/wals/a823ce0b7b174b53902a85b7f2e9f59f
I20240502 14:15:43.832814 26272 catalog_manager.cc:4830] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097): tablet a823ce0b7b174b53902a85b7f2e9f59f (table rebalancer_extra_table_44 [id=d20a7e521c7f4f93a17f7a49aa1c5d28]) successfully deleted
W20240502 14:15:44.128880 26493 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.197:38715, user_credentials={real_user=slave}} blocked reactor thread for 45321.2us
I20240502 14:15:44.401618 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:44.457593 29421 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:44.474108 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:44.497692 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:44.531514 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:44.548358 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:44.561863 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:44.593686 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:44.763296 26558 consensus_queue.cc:237] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:44.765619 29306 raft_consensus.cc:1261] T f94a6ca24bf748c1ac4f12df761d537d P c1100e16b3b9495ca61d4f95e89b6a89 [term 2 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:15:44.765982 29440 raft_consensus.cc:1261] T f94a6ca24bf748c1ac4f12df761d537d P 68a3c2b17ed94e06aba66ed4178132ab [term 2 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:15:44.766520 3312 consensus_queue.cc:1035] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:44.766808 3822 consensus_queue.cc:1035] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:44.770689 3884 raft_consensus.cc:2932] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:44.771340 29440 raft_consensus.cc:2932] T f94a6ca24bf748c1ac4f12df761d537d P 68a3c2b17ed94e06aba66ed4178132ab [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:44.771277 29306 raft_consensus.cc:2932] T f94a6ca24bf748c1ac4f12df761d537d P c1100e16b3b9495ca61d4f95e89b6a89 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:44.777688 26285 catalog_manager.cc:5484] T f94a6ca24bf748c1ac4f12df761d537d P 68a3c2b17ed94e06aba66ed4178132ab reported cstate change: config changed from index 2 to 4, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) evicted. New cstate: current_term: 2 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:44.786485 29155 tablet_service.cc:1514] Processing DeleteTablet for tablet f94a6ca24bf748c1ac4f12df761d537d with delete_type TABLET_DATA_TOMBSTONED (TS 1cb67158133e4648a6eceba32d4dbba3 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55202
I20240502 14:15:44.790858 3892 tablet_replica.cc:330] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3: stopping tablet replica
I20240502 14:15:44.791103 3892 raft_consensus.cc:2229] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:44.791333 3892 raft_consensus.cc:2258] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:44.792338 3892 ts_tablet_manager.cc:1875] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:44.798134 3892 ts_tablet_manager.cc:1888] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240502 14:15:44.798256 3892 log.cc:1196] T f94a6ca24bf748c1ac4f12df761d537d P 1cb67158133e4648a6eceba32d4dbba3: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal/wals/f94a6ca24bf748c1ac4f12df761d537d
I20240502 14:15:44.798985 26271 catalog_manager.cc:4830] TS 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): tablet f94a6ca24bf748c1ac4f12df761d537d (table rebalancer_extra_table_10 [id=77b7f84f16494751ad1f1e50fa6a8db0]) successfully deleted
W20240502 14:15:45.127519 29111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.194:36025, user_credentials={real_user=slave}} blocked reactor thread for 45627.4us
I20240502 14:15:45.181020 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:45.186002 29421 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:45.195602 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:45.214738 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:45.226541 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:45.359865 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:45.365499 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:45.423238 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 14:15:45.464820 29510 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.193:37985, user_credentials={real_user=slave}} blocked reactor thread for 110021us
I20240502 14:15:45.576745 29174 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c32ea761f2834d758529890e5b496f88"
dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36020
I20240502 14:15:45.576956 29174 raft_consensus.cc:595] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:15:45.752842 3841 raft_consensus.cc:979] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3: : Instructing follower 68a3c2b17ed94e06aba66ed4178132ab to start an election
I20240502 14:15:45.753787 3995 raft_consensus.cc:1067] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 LEADER]: Signalling peer 68a3c2b17ed94e06aba66ed4178132ab to start an election
I20240502 14:15:45.754853 29439 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c32ea761f2834d758529890e5b496f88"
dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
from {username='slave'} at 127.19.94.197:58923
I20240502 14:15:45.755139 29439 raft_consensus.cc:482] T c32ea761f2834d758529890e5b496f88 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:45.755265 29439 raft_consensus.cc:3037] T c32ea761f2834d758529890e5b496f88 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:45.761896 29439 raft_consensus.cc:504] T c32ea761f2834d758529890e5b496f88 P 68a3c2b17ed94e06aba66ed4178132ab [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:45.765626 26816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32ea761f2834d758529890e5b496f88" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:15:45.765853 26816 raft_consensus.cc:3037] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:45.773746 26816 raft_consensus.cc:2454] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 2.
I20240502 14:15:45.775732 29176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32ea761f2834d758529890e5b496f88" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:15:45.776329 29307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32ea761f2834d758529890e5b496f88" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
I20240502 14:15:45.776628 29307 raft_consensus.cc:3037] T c32ea761f2834d758529890e5b496f88 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:45.780115 29307 raft_consensus.cc:2454] T c32ea761f2834d758529890e5b496f88 P c1100e16b3b9495ca61d4f95e89b6a89 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 2.
I20240502 14:15:45.790686 29378 leader_election.cc:302] T c32ea761f2834d758529890e5b496f88 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:15:45.791271 3513 raft_consensus.cc:2781] T c32ea761f2834d758529890e5b496f88 P 68a3c2b17ed94e06aba66ed4178132ab [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:45.811579 29439 leader_election.cc:288] T c32ea761f2834d758529890e5b496f88 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 2 election: Requested vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:15:45.814888 29176 raft_consensus.cc:3032] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:45.815054 29176 raft_consensus.cc:729] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Running, Role: LEADER
I20240502 14:15:45.815317 29176 consensus_queue.cc:260] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:45.815963 29176 raft_consensus.cc:3037] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:45.819316 29176 raft_consensus.cc:2454] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 2.
I20240502 14:15:45.841145 3513 raft_consensus.cc:686] T c32ea761f2834d758529890e5b496f88 P 68a3c2b17ed94e06aba66ed4178132ab [term 2 LEADER]: Becoming Leader. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Running, Role: LEADER
I20240502 14:15:45.841660 3513 consensus_queue.cc:237] T c32ea761f2834d758529890e5b496f88 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:45.853638 26277 catalog_manager.cc:5484] T c32ea761f2834d758529890e5b496f88 P 68a3c2b17ed94e06aba66ed4178132ab reported cstate change: term changed from 1 to 2, leader changed from 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) to 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199). New cstate: current_term: 2 leader_uuid: "68a3c2b17ed94e06aba66ed4178132ab" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:15:45.981673 29155 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:45.994346 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:46.022029 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:46.080377 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:46.097954 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:15:46.137322 29511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.199:41781, user_credentials={real_user=slave}} blocked reactor thread for 100183us
I20240502 14:15:46.169361 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:46.186616 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:46.227331 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:46.239568 29174 raft_consensus.cc:1261] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:15:46.240278 3513 consensus_queue.cc:1035] T c32ea761f2834d758529890e5b496f88 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:46.289602 29307 raft_consensus.cc:1261] T c32ea761f2834d758529890e5b496f88 P c1100e16b3b9495ca61d4f95e89b6a89 [term 2 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:15:46.290632 3513 consensus_queue.cc:1035] T c32ea761f2834d758529890e5b496f88 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:46.302153 26818 raft_consensus.cc:1261] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:15:46.302919 4000 consensus_queue.cc:1035] T c32ea761f2834d758529890e5b496f88 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:46.556802 29439 consensus_queue.cc:237] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
W20240502 14:15:46.559428 29379 consensus_peers.cc:485] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): Couldn't send request to peer e4cfb8fc5a474177af6848a6efa13a27. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4cea6495041642f58e1eb5d26fc10df5. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:46.559623 26425 raft_consensus.cc:1261] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:46.560213 3513 consensus_queue.cc:1035] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:46.560384 29307 raft_consensus.cc:1261] T 4cea6495041642f58e1eb5d26fc10df5 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:46.561077 3513 consensus_queue.cc:1035] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:46.565279 3999 raft_consensus.cc:2932] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:46.565886 26425 raft_consensus.cc:2932] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:46.575109 29307 raft_consensus.cc:2932] T 4cea6495041642f58e1eb5d26fc10df5 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:46.576206 26279 catalog_manager.cc:5484] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab reported cstate change: config changed from index -1 to 2, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New cstate: current_term: 1 leader_uuid: "68a3c2b17ed94e06aba66ed4178132ab" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:46.652815 29174 consensus_queue.cc:237] T f6561a18519245f9a99690259c9c295e P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:46.655419 26818 raft_consensus.cc:1261] T f6561a18519245f9a99690259c9c295e P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:15:46.655575 29112 consensus_peers.cc:485] T f6561a18519245f9a99690259c9c295e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): Couldn't send request to peer 69ab78b2a1b74d2bbf2cdeafa0a4af23. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f6561a18519245f9a99690259c9c295e. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:46.655577 29573 raft_consensus.cc:1261] T f6561a18519245f9a99690259c9c295e P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:46.656152 4125 consensus_queue.cc:1035] T f6561a18519245f9a99690259c9c295e P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:46.656543 3995 consensus_queue.cc:1035] T f6561a18519245f9a99690259c9c295e P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:46.672302 3995 raft_consensus.cc:2932] T f6561a18519245f9a99690259c9c295e P 1cb67158133e4648a6eceba32d4dbba3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:46.673101 26816 raft_consensus.cc:2932] T f6561a18519245f9a99690259c9c295e P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:46.679486 26278 catalog_manager.cc:5484] T f6561a18519245f9a99690259c9c295e P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New cstate: current_term: 1 leader_uuid: "1cb67158133e4648a6eceba32d4dbba3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:46.681928 29573 raft_consensus.cc:2932] T f6561a18519245f9a99690259c9c295e P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:46.732098 26689 consensus_queue.cc:237] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
W20240502 14:15:46.735119 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c41522c60f3942aa9f28f514dcad4e49. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:46.738113 26425 raft_consensus.cc:1261] T c41522c60f3942aa9f28f514dcad4e49 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:46.738323 26818 raft_consensus.cc:1261] T c41522c60f3942aa9f28f514dcad4e49 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:46.738823 3441 consensus_queue.cc:1035] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:46.739454 3585 consensus_queue.cc:1035] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:46.756336 3585 raft_consensus.cc:2932] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:15:46.757143 26427 raft_consensus.cc:2932] T c41522c60f3942aa9f28f514dcad4e49 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:15:46.764106 26277 catalog_manager.cc:5484] T c41522c60f3942aa9f28f514dcad4e49 P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: config changed from index -1 to 4, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:15:46.787478 26818 raft_consensus.cc:2932] T c41522c60f3942aa9f28f514dcad4e49 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:15:46.834578 26425 consensus_queue.cc:237] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:46.837327 29307 raft_consensus.cc:1261] T 822e90c09531419e9b32af0438fceb57 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:15:46.837633 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): Couldn't send request to peer e4cfb8fc5a474177af6848a6efa13a27. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 822e90c09531419e9b32af0438fceb57. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:46.837694 29439 raft_consensus.cc:1261] T 822e90c09531419e9b32af0438fceb57 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:46.838104 4167 consensus_queue.cc:1035] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:46.838439 3915 consensus_queue.cc:1035] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:46.844825 3915 raft_consensus.cc:2932] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:46.849133 29307 raft_consensus.cc:2932] T 822e90c09531419e9b32af0438fceb57 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:46.849570 29439 raft_consensus.cc:2932] T 822e90c09531419e9b32af0438fceb57 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:46.859350 26278 catalog_manager.cc:5484] T 822e90c09531419e9b32af0438fceb57 P 68a3c2b17ed94e06aba66ed4178132ab reported cstate change: config changed from index -1 to 2, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:46.901350 29439 consensus_queue.cc:237] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:46.904261 29174 raft_consensus.cc:1261] T 457516f59dee4436b474af63819fad4e P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:46.904238 29307 raft_consensus.cc:1261] T 457516f59dee4436b474af63819fad4e P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:15:46.904820 29378 consensus_peers.cc:485] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab -> Peer ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): Couldn't send request to peer ba187e38a2754d60a12de9ac4856430a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 457516f59dee4436b474af63819fad4e. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:46.905164 4186 consensus_queue.cc:1035] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:46.905460 3513 consensus_queue.cc:1035] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:46.910351 3513 raft_consensus.cc:2932] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:46.911712 29174 raft_consensus.cc:2932] T 457516f59dee4436b474af63819fad4e P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:46.917147 29307 raft_consensus.cc:2932] T 457516f59dee4436b474af63819fad4e P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:46.919626 26279 catalog_manager.cc:5484] T 457516f59dee4436b474af63819fad4e P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New cstate: current_term: 1 leader_uuid: "68a3c2b17ed94e06aba66ed4178132ab" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:46.977984 26687 consensus_queue.cc:237] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:46.981127 26425 raft_consensus.cc:1261] T 220adc4ff7dc4a3b957e479da22fbd3d P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:46.981617 29174 raft_consensus.cc:1261] T 220adc4ff7dc4a3b957e479da22fbd3d P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:46.981900 3585 consensus_queue.cc:1035] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:15:46.982255 26625 consensus_peers.cc:485] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097): Couldn't send request to peer 4092902b3e674857a0147f1b4f2cdb86. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 220adc4ff7dc4a3b957e479da22fbd3d. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:46.982281 4149 consensus_queue.cc:1035] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:46.993952 4149 raft_consensus.cc:2932] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:15:46.995616 29174 raft_consensus.cc:2932] T 220adc4ff7dc4a3b957e479da22fbd3d P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:15:46.998586 26425 raft_consensus.cc:2932] T 220adc4ff7dc4a3b957e479da22fbd3d P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:15:47.001421 26279 catalog_manager.cc:5484] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: config changed from index -1 to 2, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:15:47.080881 4224 ts_tablet_manager.cc:902] T f6561a18519245f9a99690259c9c295e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Initiating tablet copy from peer 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:15:47.087800 29307 consensus_queue.cc:237] T 3f6d55d89e55425183da73dbb3f4175f P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:47.098472 26555 raft_consensus.cc:1261] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer c1100e16b3b9495ca61d4f95e89b6a89: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:15:47.098678 29244 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P c1100e16b3b9495ca61d4f95e89b6a89 -> Peer 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): Couldn't send request to peer 89af7f423cf44717a08d6b142f0ed49d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3f6d55d89e55425183da73dbb3f4175f. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:47.099294 4064 consensus_queue.cc:1035] T 3f6d55d89e55425183da73dbb3f4175f P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:47.101475 29439 raft_consensus.cc:1261] T 3f6d55d89e55425183da73dbb3f4175f P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer c1100e16b3b9495ca61d4f95e89b6a89: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:47.106532 4228 consensus_queue.cc:1035] T 3f6d55d89e55425183da73dbb3f4175f P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:47.121881 4224 tablet_copy_client.cc:308] T f6561a18519245f9a99690259c9c295e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.197:38715
I20240502 14:15:47.122473 4228 raft_consensus.cc:2932] T 3f6d55d89e55425183da73dbb3f4175f P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:47.124122 29196 tablet_copy_service.cc:140] P 1cb67158133e4648a6eceba32d4dbba3: Received BeginTabletCopySession request for tablet f6561a18519245f9a99690259c9c295e from peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 ({username='slave'} at 127.19.94.195:56139)
I20240502 14:15:47.124292 29196 tablet_copy_service.cc:161] P 1cb67158133e4648a6eceba32d4dbba3: Beginning new tablet copy session on tablet f6561a18519245f9a99690259c9c295e from peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 at {username='slave'} at 127.19.94.195:56139: session id = 69ab78b2a1b74d2bbf2cdeafa0a4af23-f6561a18519245f9a99690259c9c295e
I20240502 14:15:47.126787 4234 ts_tablet_manager.cc:902] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27: Initiating tablet copy from peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:15:47.126797 26555 raft_consensus.cc:2932] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:47.127974 29196 tablet_copy_source_session.cc:205] T f6561a18519245f9a99690259c9c295e P 1cb67158133e4648a6eceba32d4dbba3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:47.133719 26277 catalog_manager.cc:5484] T 3f6d55d89e55425183da73dbb3f4175f P c1100e16b3b9495ca61d4f95e89b6a89 reported cstate change: config changed from index -1 to 2, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New cstate: current_term: 1 leader_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:47.136663 4224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6561a18519245f9a99690259c9c295e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:47.142280 29439 raft_consensus.cc:2932] T 3f6d55d89e55425183da73dbb3f4175f P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:47.168848 4234 tablet_copy_client.cc:308] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.199:41781
I20240502 14:15:47.169606 29463 tablet_copy_service.cc:140] P 68a3c2b17ed94e06aba66ed4178132ab: Received BeginTabletCopySession request for tablet 4cea6495041642f58e1eb5d26fc10df5 from peer e4cfb8fc5a474177af6848a6efa13a27 ({username='slave'} at 127.19.94.196:41215)
I20240502 14:15:47.169785 29463 tablet_copy_service.cc:161] P 68a3c2b17ed94e06aba66ed4178132ab: Beginning new tablet copy session on tablet 4cea6495041642f58e1eb5d26fc10df5 from peer e4cfb8fc5a474177af6848a6efa13a27 at {username='slave'} at 127.19.94.196:41215: session id = e4cfb8fc5a474177af6848a6efa13a27-4cea6495041642f58e1eb5d26fc10df5
I20240502 14:15:47.173134 4224 tablet_copy_client.cc:783] T f6561a18519245f9a99690259c9c295e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:47.173319 29463 tablet_copy_source_session.cc:205] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:47.173447 4224 tablet_copy_client.cc:647] T f6561a18519245f9a99690259c9c295e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:47.175427 4224 tablet_copy_client.cc:523] T f6561a18519245f9a99690259c9c295e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:47.179759 4234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cea6495041642f58e1eb5d26fc10df5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:47.180752 4224 tablet_bootstrap.cc:492] T f6561a18519245f9a99690259c9c295e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:15:47.187721 4224 tablet_bootstrap.cc:492] T f6561a18519245f9a99690259c9c295e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:47.188134 4224 tablet_bootstrap.cc:492] T f6561a18519245f9a99690259c9c295e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap complete.
I20240502 14:15:47.188432 4224 ts_tablet_manager.cc:1367] T f6561a18519245f9a99690259c9c295e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.001s
I20240502 14:15:47.189146 4224 raft_consensus.cc:348] T f6561a18519245f9a99690259c9c295e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:47.189390 4224 raft_consensus.cc:729] T f6561a18519245f9a99690259c9c295e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:15:47.189656 4224 consensus_queue.cc:260] T f6561a18519245f9a99690259c9c295e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:47.190343 4224 ts_tablet_manager.cc:1398] T f6561a18519245f9a99690259c9c295e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:47.191052 4234 tablet_copy_client.cc:783] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:47.191159 29196 tablet_copy_service.cc:342] P 1cb67158133e4648a6eceba32d4dbba3: Request end of tablet copy session 69ab78b2a1b74d2bbf2cdeafa0a4af23-f6561a18519245f9a99690259c9c295e received from {username='slave'} at 127.19.94.195:56139
I20240502 14:15:47.191303 29196 tablet_copy_service.cc:433] P 1cb67158133e4648a6eceba32d4dbba3: ending tablet copy session 69ab78b2a1b74d2bbf2cdeafa0a4af23-f6561a18519245f9a99690259c9c295e on tablet f6561a18519245f9a99690259c9c295e with peer 69ab78b2a1b74d2bbf2cdeafa0a4af23
I20240502 14:15:47.191303 4234 tablet_copy_client.cc:647] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:47.196503 4234 tablet_copy_client.cc:523] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:47.201716 4234 tablet_bootstrap.cc:492] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:15:47.220257 4234 tablet_bootstrap.cc:492] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:47.220703 4234 tablet_bootstrap.cc:492] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap complete.
I20240502 14:15:47.220974 4234 ts_tablet_manager.cc:1367] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.011s
I20240502 14:15:47.221649 4234 raft_consensus.cc:348] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:47.221879 4234 raft_consensus.cc:729] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:15:47.222074 4234 consensus_queue.cc:260] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:47.222780 4234 ts_tablet_manager.cc:1398] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:15:47.223827 29463 tablet_copy_service.cc:342] P 68a3c2b17ed94e06aba66ed4178132ab: Request end of tablet copy session e4cfb8fc5a474177af6848a6efa13a27-4cea6495041642f58e1eb5d26fc10df5 received from {username='slave'} at 127.19.94.196:41215
I20240502 14:15:47.223965 29463 tablet_copy_service.cc:433] P 68a3c2b17ed94e06aba66ed4178132ab: ending tablet copy session e4cfb8fc5a474177af6848a6efa13a27-4cea6495041642f58e1eb5d26fc10df5 on tablet 4cea6495041642f58e1eb5d26fc10df5 with peer e4cfb8fc5a474177af6848a6efa13a27
I20240502 14:15:47.261042 4252 ts_tablet_manager.cc:902] T c41522c60f3942aa9f28f514dcad4e49 P 68a3c2b17ed94e06aba66ed4178132ab: Initiating tablet copy from peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:15:47.279513 4252 tablet_copy_client.cc:308] T c41522c60f3942aa9f28f514dcad4e49 P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.195:44325
I20240502 14:15:47.280292 26709 tablet_copy_service.cc:140] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Received BeginTabletCopySession request for tablet c41522c60f3942aa9f28f514dcad4e49 from peer 68a3c2b17ed94e06aba66ed4178132ab ({username='slave'} at 127.19.94.199:51755)
I20240502 14:15:47.280450 26709 tablet_copy_service.cc:161] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Beginning new tablet copy session on tablet c41522c60f3942aa9f28f514dcad4e49 from peer 68a3c2b17ed94e06aba66ed4178132ab at {username='slave'} at 127.19.94.199:51755: session id = 68a3c2b17ed94e06aba66ed4178132ab-c41522c60f3942aa9f28f514dcad4e49
I20240502 14:15:47.283353 26709 tablet_copy_source_session.cc:205] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:47.284937 4252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c41522c60f3942aa9f28f514dcad4e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:47.291313 4252 tablet_copy_client.cc:783] T c41522c60f3942aa9f28f514dcad4e49 P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:47.291584 4252 tablet_copy_client.cc:647] T c41522c60f3942aa9f28f514dcad4e49 P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:47.293453 4252 tablet_copy_client.cc:523] T c41522c60f3942aa9f28f514dcad4e49 P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:47.294255 26425 consensus_queue.cc:237] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
W20240502 14:15:47.296954 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6814613d36a74353a99673cf16aa9821. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:47.297039 26555 raft_consensus.cc:1261] T 6814613d36a74353a99673cf16aa9821 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:47.297133 26818 raft_consensus.cc:1261] T 6814613d36a74353a99673cf16aa9821 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:47.297338 4252 tablet_bootstrap.cc:492] T c41522c60f3942aa9f28f514dcad4e49 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:15:47.297592 4167 consensus_queue.cc:1035] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:47.297888 3816 consensus_queue.cc:1035] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:47.301920 4167 raft_consensus.cc:2932] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:15:47.302593 26818 raft_consensus.cc:2932] T 6814613d36a74353a99673cf16aa9821 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:15:47.308984 26278 catalog_manager.cc:5484] T 6814613d36a74353a99673cf16aa9821 P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: config changed from index -1 to 4, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:15:47.314512 26555 raft_consensus.cc:2932] T 6814613d36a74353a99673cf16aa9821 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:15:47.348356 4252 tablet_bootstrap.cc:492] T c41522c60f3942aa9f28f514dcad4e49 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:47.348867 4252 tablet_bootstrap.cc:492] T c41522c60f3942aa9f28f514dcad4e49 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap complete.
I20240502 14:15:47.349179 4252 ts_tablet_manager.cc:1367] T c41522c60f3942aa9f28f514dcad4e49 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.052s user 0.043s sys 0.004s
I20240502 14:15:47.350023 4252 raft_consensus.cc:348] T c41522c60f3942aa9f28f514dcad4e49 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:47.350268 4252 raft_consensus.cc:729] T c41522c60f3942aa9f28f514dcad4e49 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:15:47.350556 4252 consensus_queue.cc:260] T c41522c60f3942aa9f28f514dcad4e49 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:47.351280 4252 ts_tablet_manager.cc:1398] T c41522c60f3942aa9f28f514dcad4e49 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:15:47.352887 26709 tablet_copy_service.cc:342] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Request end of tablet copy session 68a3c2b17ed94e06aba66ed4178132ab-c41522c60f3942aa9f28f514dcad4e49 received from {username='slave'} at 127.19.94.199:51755
I20240502 14:15:47.353016 26709 tablet_copy_service.cc:433] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: ending tablet copy session 68a3c2b17ed94e06aba66ed4178132ab-c41522c60f3942aa9f28f514dcad4e49 on tablet c41522c60f3942aa9f28f514dcad4e49 with peer 68a3c2b17ed94e06aba66ed4178132ab
I20240502 14:15:47.425475 29439 consensus_queue.cc:237] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:47.429042 29306 raft_consensus.cc:1261] T e93acdb9427943938f4c12ae2bf2c594 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:15:47.429930 29379 consensus_peers.cc:485] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): Couldn't send request to peer 69ab78b2a1b74d2bbf2cdeafa0a4af23. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e93acdb9427943938f4c12ae2bf2c594. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:47.430182 29176 raft_consensus.cc:1261] T e93acdb9427943938f4c12ae2bf2c594 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:47.430980 3513 consensus_queue.cc:1035] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:47.431515 4186 consensus_queue.cc:1035] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:47.450099 4283 raft_consensus.cc:2932] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:47.450981 29176 raft_consensus.cc:2932] T e93acdb9427943938f4c12ae2bf2c594 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:47.458205 26279 catalog_manager.cc:5484] T e93acdb9427943938f4c12ae2bf2c594 P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New cstate: current_term: 1 leader_uuid: "68a3c2b17ed94e06aba66ed4178132ab" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:47.460772 29307 raft_consensus.cc:2932] T e93acdb9427943938f4c12ae2bf2c594 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:47.461488 4234 ts_tablet_manager.cc:902] T 822e90c09531419e9b32af0438fceb57 P e4cfb8fc5a474177af6848a6efa13a27: Initiating tablet copy from peer 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:15:47.480911 4290 ts_tablet_manager.cc:902] T 220adc4ff7dc4a3b957e479da22fbd3d P 4092902b3e674857a0147f1b4f2cdb86: Initiating tablet copy from peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:15:47.490842 4234 tablet_copy_client.cc:308] T 822e90c09531419e9b32af0438fceb57 P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.193:37985
I20240502 14:15:47.491771 26447 tablet_copy_service.cc:140] P 89af7f423cf44717a08d6b142f0ed49d: Received BeginTabletCopySession request for tablet 822e90c09531419e9b32af0438fceb57 from peer e4cfb8fc5a474177af6848a6efa13a27 ({username='slave'} at 127.19.94.196:59299)
I20240502 14:15:47.491917 26447 tablet_copy_service.cc:161] P 89af7f423cf44717a08d6b142f0ed49d: Beginning new tablet copy session on tablet 822e90c09531419e9b32af0438fceb57 from peer e4cfb8fc5a474177af6848a6efa13a27 at {username='slave'} at 127.19.94.196:59299: session id = e4cfb8fc5a474177af6848a6efa13a27-822e90c09531419e9b32af0438fceb57
I20240502 14:15:47.494735 26447 tablet_copy_source_session.cc:205] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:47.496356 4234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 822e90c09531419e9b32af0438fceb57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:47.498327 4290 tablet_copy_client.cc:308] T 220adc4ff7dc4a3b957e479da22fbd3d P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.195:44325
I20240502 14:15:47.500280 26709 tablet_copy_service.cc:140] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Received BeginTabletCopySession request for tablet 220adc4ff7dc4a3b957e479da22fbd3d from peer 4092902b3e674857a0147f1b4f2cdb86 ({username='slave'} at 127.19.94.200:47557)
I20240502 14:15:47.500465 26709 tablet_copy_service.cc:161] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Beginning new tablet copy session on tablet 220adc4ff7dc4a3b957e479da22fbd3d from peer 4092902b3e674857a0147f1b4f2cdb86 at {username='slave'} at 127.19.94.200:47557: session id = 4092902b3e674857a0147f1b4f2cdb86-220adc4ff7dc4a3b957e479da22fbd3d
I20240502 14:15:47.504448 4234 tablet_copy_client.cc:783] T 822e90c09531419e9b32af0438fceb57 P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:47.505038 4234 tablet_copy_client.cc:647] T 822e90c09531419e9b32af0438fceb57 P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:47.505046 26709 tablet_copy_source_session.cc:205] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:47.508368 4234 tablet_copy_client.cc:523] T 822e90c09531419e9b32af0438fceb57 P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:47.509338 4290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 220adc4ff7dc4a3b957e479da22fbd3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:47.512480 4234 tablet_bootstrap.cc:492] T 822e90c09531419e9b32af0438fceb57 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:15:47.518918 4290 tablet_copy_client.cc:783] T 220adc4ff7dc4a3b957e479da22fbd3d P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:47.519276 4290 tablet_copy_client.cc:647] T 220adc4ff7dc4a3b957e479da22fbd3d P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:47.522056 4290 tablet_copy_client.cc:523] T 220adc4ff7dc4a3b957e479da22fbd3d P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:47.530968 4234 tablet_bootstrap.cc:492] T 822e90c09531419e9b32af0438fceb57 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:47.531478 4234 tablet_bootstrap.cc:492] T 822e90c09531419e9b32af0438fceb57 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap complete.
I20240502 14:15:47.531805 4234 ts_tablet_manager.cc:1367] T 822e90c09531419e9b32af0438fceb57 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.011s
I20240502 14:15:47.532795 4234 raft_consensus.cc:348] T 822e90c09531419e9b32af0438fceb57 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:47.533105 4290 tablet_bootstrap.cc:492] T 220adc4ff7dc4a3b957e479da22fbd3d P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:15:47.533141 4234 raft_consensus.cc:729] T 822e90c09531419e9b32af0438fceb57 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:15:47.533517 4234 consensus_queue.cc:260] T 822e90c09531419e9b32af0438fceb57 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:47.534130 4234 ts_tablet_manager.cc:1398] T 822e90c09531419e9b32af0438fceb57 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:47.534796 26447 tablet_copy_service.cc:342] P 89af7f423cf44717a08d6b142f0ed49d: Request end of tablet copy session e4cfb8fc5a474177af6848a6efa13a27-822e90c09531419e9b32af0438fceb57 received from {username='slave'} at 127.19.94.196:59299
I20240502 14:15:47.534924 26447 tablet_copy_service.cc:433] P 89af7f423cf44717a08d6b142f0ed49d: ending tablet copy session e4cfb8fc5a474177af6848a6efa13a27-822e90c09531419e9b32af0438fceb57 on tablet 822e90c09531419e9b32af0438fceb57 with peer e4cfb8fc5a474177af6848a6efa13a27
I20240502 14:15:47.541126 4302 ts_tablet_manager.cc:902] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a: Initiating tablet copy from peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:15:47.547889 4308 ts_tablet_manager.cc:902] T 3f6d55d89e55425183da73dbb3f4175f P 89af7f423cf44717a08d6b142f0ed49d: Initiating tablet copy from peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:15:47.561172 4290 tablet_bootstrap.cc:492] T 220adc4ff7dc4a3b957e479da22fbd3d P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:47.561821 4290 tablet_bootstrap.cc:492] T 220adc4ff7dc4a3b957e479da22fbd3d P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap complete.
I20240502 14:15:47.562145 4290 ts_tablet_manager.cc:1367] T 220adc4ff7dc4a3b957e479da22fbd3d P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.000s
I20240502 14:15:47.563035 4290 raft_consensus.cc:348] T 220adc4ff7dc4a3b957e479da22fbd3d P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:47.563297 4290 raft_consensus.cc:729] T 220adc4ff7dc4a3b957e479da22fbd3d P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:15:47.563478 26816 raft_consensus.cc:1203] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:47.566460 4290 consensus_queue.cc:260] T 220adc4ff7dc4a3b957e479da22fbd3d P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:47.576299 4290 ts_tablet_manager.cc:1398] T 220adc4ff7dc4a3b957e479da22fbd3d P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.014s user 0.004s sys 0.000s
I20240502 14:15:47.577332 4302 tablet_copy_client.cc:308] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.199:41781
I20240502 14:15:47.578272 29463 tablet_copy_service.cc:140] P 68a3c2b17ed94e06aba66ed4178132ab: Received BeginTabletCopySession request for tablet 457516f59dee4436b474af63819fad4e from peer ba187e38a2754d60a12de9ac4856430a ({username='slave'} at 127.19.94.194:57813)
I20240502 14:15:47.578474 29463 tablet_copy_service.cc:161] P 68a3c2b17ed94e06aba66ed4178132ab: Beginning new tablet copy session on tablet 457516f59dee4436b474af63819fad4e from peer ba187e38a2754d60a12de9ac4856430a at {username='slave'} at 127.19.94.194:57813: session id = ba187e38a2754d60a12de9ac4856430a-457516f59dee4436b474af63819fad4e
I20240502 14:15:47.579347 26709 tablet_copy_service.cc:342] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Request end of tablet copy session 4092902b3e674857a0147f1b4f2cdb86-220adc4ff7dc4a3b957e479da22fbd3d received from {username='slave'} at 127.19.94.200:47557
I20240502 14:15:47.579459 26709 tablet_copy_service.cc:433] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: ending tablet copy session 4092902b3e674857a0147f1b4f2cdb86-220adc4ff7dc4a3b957e479da22fbd3d on tablet 220adc4ff7dc4a3b957e479da22fbd3d with peer 4092902b3e674857a0147f1b4f2cdb86
I20240502 14:15:47.581419 29463 tablet_copy_source_session.cc:205] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:47.582839 4302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 457516f59dee4436b474af63819fad4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:47.589794 4302 tablet_copy_client.cc:783] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:47.590132 4302 tablet_copy_client.cc:647] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:47.590682 4308 tablet_copy_client.cc:308] T 3f6d55d89e55425183da73dbb3f4175f P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.198:43659
I20240502 14:15:47.591432 29329 tablet_copy_service.cc:140] P c1100e16b3b9495ca61d4f95e89b6a89: Received BeginTabletCopySession request for tablet 3f6d55d89e55425183da73dbb3f4175f from peer 89af7f423cf44717a08d6b142f0ed49d ({username='slave'} at 127.19.94.193:58815)
I20240502 14:15:47.591583 29329 tablet_copy_service.cc:161] P c1100e16b3b9495ca61d4f95e89b6a89: Beginning new tablet copy session on tablet 3f6d55d89e55425183da73dbb3f4175f from peer 89af7f423cf44717a08d6b142f0ed49d at {username='slave'} at 127.19.94.193:58815: session id = 89af7f423cf44717a08d6b142f0ed49d-3f6d55d89e55425183da73dbb3f4175f
I20240502 14:15:47.593024 4302 tablet_copy_client.cc:523] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:47.594560 29329 tablet_copy_source_session.cc:205] T 3f6d55d89e55425183da73dbb3f4175f P c1100e16b3b9495ca61d4f95e89b6a89: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:47.596026 4308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f6d55d89e55425183da73dbb3f4175f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:47.596827 4302 tablet_bootstrap.cc:492] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:15:47.615192 4308 tablet_copy_client.cc:783] T 3f6d55d89e55425183da73dbb3f4175f P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:47.631804 4308 tablet_copy_client.cc:647] T 3f6d55d89e55425183da73dbb3f4175f P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:47.635464 4302 tablet_bootstrap.cc:492] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:47.635915 4302 tablet_bootstrap.cc:492] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a: Bootstrap complete.
I20240502 14:15:47.636191 4302 ts_tablet_manager.cc:1367] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.039s user 0.025s sys 0.010s
I20240502 14:15:47.636485 4308 tablet_copy_client.cc:523] T 3f6d55d89e55425183da73dbb3f4175f P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:47.637132 4302 raft_consensus.cc:348] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:47.637346 4302 raft_consensus.cc:729] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:15:47.637567 4302 consensus_queue.cc:260] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:47.639855 4302 ts_tablet_manager.cc:1398] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:15:47.640655 29463 tablet_copy_service.cc:342] P 68a3c2b17ed94e06aba66ed4178132ab: Request end of tablet copy session ba187e38a2754d60a12de9ac4856430a-457516f59dee4436b474af63819fad4e received from {username='slave'} at 127.19.94.194:57813
I20240502 14:15:47.640816 29463 tablet_copy_service.cc:433] P 68a3c2b17ed94e06aba66ed4178132ab: ending tablet copy session ba187e38a2754d60a12de9ac4856430a-457516f59dee4436b474af63819fad4e on tablet 457516f59dee4436b474af63819fad4e with peer ba187e38a2754d60a12de9ac4856430a
I20240502 14:15:47.642338 4308 tablet_bootstrap.cc:492] T 3f6d55d89e55425183da73dbb3f4175f P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:15:47.657732 29442 raft_consensus.cc:1203] T c41522c60f3942aa9f28f514dcad4e49 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:15:47.666783 4308 tablet_bootstrap.cc:492] T 3f6d55d89e55425183da73dbb3f4175f P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:47.667341 4308 tablet_bootstrap.cc:492] T 3f6d55d89e55425183da73dbb3f4175f P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap complete.
I20240502 14:15:47.667840 4308 ts_tablet_manager.cc:1367] T 3f6d55d89e55425183da73dbb3f4175f P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.011s
I20240502 14:15:47.669139 4308 raft_consensus.cc:348] T 3f6d55d89e55425183da73dbb3f4175f P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:47.669490 4308 raft_consensus.cc:729] T 3f6d55d89e55425183da73dbb3f4175f P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:15:47.669763 4308 consensus_queue.cc:260] T 3f6d55d89e55425183da73dbb3f4175f P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:47.670539 4308 ts_tablet_manager.cc:1398] T 3f6d55d89e55425183da73dbb3f4175f P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:15:47.672132 29329 tablet_copy_service.cc:342] P c1100e16b3b9495ca61d4f95e89b6a89: Request end of tablet copy session 89af7f423cf44717a08d6b142f0ed49d-3f6d55d89e55425183da73dbb3f4175f received from {username='slave'} at 127.19.94.193:58815
I20240502 14:15:47.672287 29329 tablet_copy_service.cc:433] P c1100e16b3b9495ca61d4f95e89b6a89: ending tablet copy session 89af7f423cf44717a08d6b142f0ed49d-3f6d55d89e55425183da73dbb3f4175f on tablet 3f6d55d89e55425183da73dbb3f4175f with peer 89af7f423cf44717a08d6b142f0ed49d
I20240502 14:15:47.675055 29442 consensus_queue.cc:237] T 01e6bc3689db46e2ad7d0484a29af287 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:47.678822 29573 raft_consensus.cc:1261] T 01e6bc3689db46e2ad7d0484a29af287 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:47.678953 29176 raft_consensus.cc:1261] T 01e6bc3689db46e2ad7d0484a29af287 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:47.679597 4282 consensus_queue.cc:1035] T 01e6bc3689db46e2ad7d0484a29af287 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:47.679956 4189 consensus_queue.cc:1035] T 01e6bc3689db46e2ad7d0484a29af287 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:15:47.680449 29378 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): Couldn't send request to peer ba187e38a2754d60a12de9ac4856430a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01e6bc3689db46e2ad7d0484a29af287. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:47.683251 4189 raft_consensus.cc:2932] T 01e6bc3689db46e2ad7d0484a29af287 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:47.683921 29573 raft_consensus.cc:2932] T 01e6bc3689db46e2ad7d0484a29af287 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:47.690502 26277 catalog_manager.cc:5484] T 01e6bc3689db46e2ad7d0484a29af287 P 68a3c2b17ed94e06aba66ed4178132ab reported cstate change: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New cstate: current_term: 1 leader_uuid: "68a3c2b17ed94e06aba66ed4178132ab" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:47.692103 29176 raft_consensus.cc:2932] T 01e6bc3689db46e2ad7d0484a29af287 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:47.699867 26687 raft_consensus.cc:1203] T f6561a18519245f9a99690259c9c295e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:47.740538 4334 ts_tablet_manager.cc:902] T 6814613d36a74353a99673cf16aa9821 P c1100e16b3b9495ca61d4f95e89b6a89: Initiating tablet copy from peer 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:15:47.753237 4334 tablet_copy_client.cc:308] T 6814613d36a74353a99673cf16aa9821 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.193:37985
I20240502 14:15:47.754189 26447 tablet_copy_service.cc:140] P 89af7f423cf44717a08d6b142f0ed49d: Received BeginTabletCopySession request for tablet 6814613d36a74353a99673cf16aa9821 from peer c1100e16b3b9495ca61d4f95e89b6a89 ({username='slave'} at 127.19.94.198:52561)
I20240502 14:15:47.754369 26447 tablet_copy_service.cc:161] P 89af7f423cf44717a08d6b142f0ed49d: Beginning new tablet copy session on tablet 6814613d36a74353a99673cf16aa9821 from peer c1100e16b3b9495ca61d4f95e89b6a89 at {username='slave'} at 127.19.94.198:52561: session id = c1100e16b3b9495ca61d4f95e89b6a89-6814613d36a74353a99673cf16aa9821
I20240502 14:15:47.757386 26447 tablet_copy_source_session.cc:205] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:47.761256 4334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6814613d36a74353a99673cf16aa9821. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:47.774545 4334 tablet_copy_client.cc:783] T 6814613d36a74353a99673cf16aa9821 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:47.774808 4334 tablet_copy_client.cc:647] T 6814613d36a74353a99673cf16aa9821 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:47.777109 4334 tablet_copy_client.cc:523] T 6814613d36a74353a99673cf16aa9821 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:47.780259 4334 tablet_bootstrap.cc:492] T 6814613d36a74353a99673cf16aa9821 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:15:47.810860 29573 consensus_queue.cc:237] T c0d152eda1384b6792246b2f10d667da P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:47.814471 29442 raft_consensus.cc:1261] T c0d152eda1384b6792246b2f10d667da P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:47.815711 26555 raft_consensus.cc:1261] T c0d152eda1384b6792246b2f10d667da P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:15:47.816121 29510 consensus_peers.cc:485] T c0d152eda1384b6792246b2f10d667da P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): Couldn't send request to peer 89af7f423cf44717a08d6b142f0ed49d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0d152eda1384b6792246b2f10d667da. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:47.816758 3731 consensus_queue.cc:1035] T c0d152eda1384b6792246b2f10d667da P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:47.817397 3731 consensus_queue.cc:1035] T c0d152eda1384b6792246b2f10d667da P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:47.821377 3731 raft_consensus.cc:2932] T c0d152eda1384b6792246b2f10d667da P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:47.822026 29442 raft_consensus.cc:2932] T c0d152eda1384b6792246b2f10d667da P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:47.826289 4334 tablet_bootstrap.cc:492] T 6814613d36a74353a99673cf16aa9821 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:47.826695 4334 tablet_bootstrap.cc:492] T 6814613d36a74353a99673cf16aa9821 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap complete.
I20240502 14:15:47.826982 4334 ts_tablet_manager.cc:1367] T 6814613d36a74353a99673cf16aa9821 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.047s user 0.039s sys 0.008s
I20240502 14:15:47.827746 4334 raft_consensus.cc:348] T 6814613d36a74353a99673cf16aa9821 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:47.827957 4334 raft_consensus.cc:729] T 6814613d36a74353a99673cf16aa9821 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:15:47.828256 4334 consensus_queue.cc:260] T 6814613d36a74353a99673cf16aa9821 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:47.828197 26279 catalog_manager.cc:5484] T c0d152eda1384b6792246b2f10d667da P 68a3c2b17ed94e06aba66ed4178132ab reported cstate change: config changed from index -1 to 4, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:47.828845 4334 ts_tablet_manager.cc:1398] T 6814613d36a74353a99673cf16aa9821 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 14:15:47.829936 26555 raft_consensus.cc:2932] T c0d152eda1384b6792246b2f10d667da P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:47.831158 26447 tablet_copy_service.cc:342] P 89af7f423cf44717a08d6b142f0ed49d: Request end of tablet copy session c1100e16b3b9495ca61d4f95e89b6a89-6814613d36a74353a99673cf16aa9821 received from {username='slave'} at 127.19.94.198:52561
I20240502 14:15:47.831284 26447 tablet_copy_service.cc:433] P 89af7f423cf44717a08d6b142f0ed49d: ending tablet copy session c1100e16b3b9495ca61d4f95e89b6a89-6814613d36a74353a99673cf16aa9821 on tablet 6814613d36a74353a99673cf16aa9821 with peer c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:15:47.837800 4356 ts_tablet_manager.cc:902] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Initiating tablet copy from peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:15:47.854390 4356 tablet_copy_client.cc:308] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.199:41781
I20240502 14:15:47.855247 29463 tablet_copy_service.cc:140] P 68a3c2b17ed94e06aba66ed4178132ab: Received BeginTabletCopySession request for tablet e93acdb9427943938f4c12ae2bf2c594 from peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 ({username='slave'} at 127.19.94.195:40781)
I20240502 14:15:47.855464 29463 tablet_copy_service.cc:161] P 68a3c2b17ed94e06aba66ed4178132ab: Beginning new tablet copy session on tablet e93acdb9427943938f4c12ae2bf2c594 from peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 at {username='slave'} at 127.19.94.195:40781: session id = 69ab78b2a1b74d2bbf2cdeafa0a4af23-e93acdb9427943938f4c12ae2bf2c594
I20240502 14:15:47.858067 29463 tablet_copy_source_session.cc:205] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:47.859364 4356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e93acdb9427943938f4c12ae2bf2c594. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:47.865571 4356 tablet_copy_client.cc:783] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:47.865798 4356 tablet_copy_client.cc:647] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:47.867800 4356 tablet_copy_client.cc:523] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:47.870421 4356 tablet_bootstrap.cc:492] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:15:47.875839 4356 tablet_bootstrap.cc:492] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:47.876189 4356 tablet_bootstrap.cc:492] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap complete.
I20240502 14:15:47.876425 4356 ts_tablet_manager.cc:1367] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.004s
I20240502 14:15:47.877238 4356 raft_consensus.cc:348] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:47.877461 4356 raft_consensus.cc:729] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:15:47.877725 4356 consensus_queue.cc:260] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:47.878429 4356 ts_tablet_manager.cc:1398] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:47.882916 29463 tablet_copy_service.cc:342] P 68a3c2b17ed94e06aba66ed4178132ab: Request end of tablet copy session 69ab78b2a1b74d2bbf2cdeafa0a4af23-e93acdb9427943938f4c12ae2bf2c594 received from {username='slave'} at 127.19.94.195:40781
I20240502 14:15:47.883073 29463 tablet_copy_service.cc:433] P 68a3c2b17ed94e06aba66ed4178132ab: ending tablet copy session 69ab78b2a1b74d2bbf2cdeafa0a4af23-e93acdb9427943938f4c12ae2bf2c594 on tablet e93acdb9427943938f4c12ae2bf2c594 with peer 69ab78b2a1b74d2bbf2cdeafa0a4af23
I20240502 14:15:47.900282 26687 consensus_queue.cc:237] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:47.902590 26818 raft_consensus.cc:1261] T 1fb1ef75829641a89c4cac539f431d80 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:47.903257 26425 raft_consensus.cc:1261] T 1fb1ef75829641a89c4cac539f431d80 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:47.903225 3585 consensus_queue.cc:1035] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:15:47.903702 26624 consensus_peers.cc:485] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): Couldn't send request to peer 1cb67158133e4648a6eceba32d4dbba3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1fb1ef75829641a89c4cac539f431d80. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:47.904057 4149 consensus_queue.cc:1035] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:47.910459 4149 raft_consensus.cc:2932] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:47.913769 26818 raft_consensus.cc:2932] T 1fb1ef75829641a89c4cac539f431d80 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:47.914983 26425 raft_consensus.cc:2932] T 1fb1ef75829641a89c4cac539f431d80 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:47.917174 26278 catalog_manager.cc:5484] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: config changed from index -1 to 4, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:47.960669 29174 consensus_queue.cc:237] T 929a85cbe95746b798a7510b3a1cc19e P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:47.963780 26818 raft_consensus.cc:1261] T 929a85cbe95746b798a7510b3a1cc19e P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:47.964224 29306 raft_consensus.cc:1261] T 929a85cbe95746b798a7510b3a1cc19e P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:15:47.964494 29112 consensus_peers.cc:485] T 929a85cbe95746b798a7510b3a1cc19e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): Couldn't send request to peer 69ab78b2a1b74d2bbf2cdeafa0a4af23. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 929a85cbe95746b798a7510b3a1cc19e. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:47.964887 3995 consensus_queue.cc:1035] T 929a85cbe95746b798a7510b3a1cc19e P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:47.965144 4330 consensus_queue.cc:1035] T 929a85cbe95746b798a7510b3a1cc19e P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:47.973590 4400 raft_consensus.cc:2932] T 929a85cbe95746b798a7510b3a1cc19e P 1cb67158133e4648a6eceba32d4dbba3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:47.974215 26818 raft_consensus.cc:2932] T 929a85cbe95746b798a7510b3a1cc19e P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:47.980226 29306 raft_consensus.cc:2932] T 929a85cbe95746b798a7510b3a1cc19e P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:47.987443 26278 catalog_manager.cc:5484] T 929a85cbe95746b798a7510b3a1cc19e P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New cstate: current_term: 1 leader_uuid: "1cb67158133e4648a6eceba32d4dbba3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:48.002686 26425 raft_consensus.cc:1203] T 3f6d55d89e55425183da73dbb3f4175f P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:48.024264 26818 raft_consensus.cc:1203] T 822e90c09531419e9b32af0438fceb57 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240502 14:15:48.034391 29245 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.199:41781, user_credentials={real_user=slave}} blocked reactor thread for 56638us
I20240502 14:15:48.071727 29574 raft_consensus.cc:1203] T 220adc4ff7dc4a3b957e479da22fbd3d P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:48.094342 4302 ts_tablet_manager.cc:902] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a: Initiating tablet copy from peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:15:48.099179 29306 consensus_queue.cc:237] T d6c8fa23fffe491fbfcec03afdb2052a P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:48.111135 26687 raft_consensus.cc:1261] T d6c8fa23fffe491fbfcec03afdb2052a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer c1100e16b3b9495ca61d4f95e89b6a89: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:48.111660 29574 raft_consensus.cc:1261] T d6c8fa23fffe491fbfcec03afdb2052a P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Refusing update from remote peer c1100e16b3b9495ca61d4f95e89b6a89: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:48.112046 4403 consensus_queue.cc:1035] T d6c8fa23fffe491fbfcec03afdb2052a P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:15:48.113150 29244 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P c1100e16b3b9495ca61d4f95e89b6a89 -> Peer 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): Couldn't send request to peer 89af7f423cf44717a08d6b142f0ed49d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d6c8fa23fffe491fbfcec03afdb2052a. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:48.121758 26558 raft_consensus.cc:1203] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:48.122718 4227 consensus_queue.cc:1035] T d6c8fa23fffe491fbfcec03afdb2052a P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:48.135103 29306 raft_consensus.cc:1203] T 6814613d36a74353a99673cf16aa9821 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:15:48.148463 4227 raft_consensus.cc:2932] T d6c8fa23fffe491fbfcec03afdb2052a P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:48.161533 29574 raft_consensus.cc:2932] T d6c8fa23fffe491fbfcec03afdb2052a P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:48.168522 26279 catalog_manager.cc:5484] T d6c8fa23fffe491fbfcec03afdb2052a P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: config changed from index -1 to 2, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New cstate: current_term: 1 leader_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:48.178498 4302 tablet_copy_client.cc:308] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.199:41781
I20240502 14:15:48.178972 26687 raft_consensus.cc:2932] T d6c8fa23fffe491fbfcec03afdb2052a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:48.181377 29463 tablet_copy_service.cc:140] P 68a3c2b17ed94e06aba66ed4178132ab: Received BeginTabletCopySession request for tablet 01e6bc3689db46e2ad7d0484a29af287 from peer ba187e38a2754d60a12de9ac4856430a ({username='slave'} at 127.19.94.194:57813)
I20240502 14:15:48.181591 29463 tablet_copy_service.cc:161] P 68a3c2b17ed94e06aba66ed4178132ab: Beginning new tablet copy session on tablet 01e6bc3689db46e2ad7d0484a29af287 from peer ba187e38a2754d60a12de9ac4856430a at {username='slave'} at 127.19.94.194:57813: session id = ba187e38a2754d60a12de9ac4856430a-01e6bc3689db46e2ad7d0484a29af287
I20240502 14:15:48.194314 29463 tablet_copy_source_session.cc:205] T 01e6bc3689db46e2ad7d0484a29af287 P 68a3c2b17ed94e06aba66ed4178132ab: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:48.198036 4302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01e6bc3689db46e2ad7d0484a29af287. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:48.205672 4302 tablet_copy_client.cc:783] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:48.205950 4302 tablet_copy_client.cc:647] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:48.208701 4302 tablet_copy_client.cc:523] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:48.210028 4431 ts_tablet_manager.cc:902] T c0d152eda1384b6792246b2f10d667da P 89af7f423cf44717a08d6b142f0ed49d: Initiating tablet copy from peer 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:15:48.217028 4302 tablet_bootstrap.cc:492] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:15:48.230443 26687 raft_consensus.cc:1203] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:48.234176 4302 tablet_bootstrap.cc:492] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:48.234694 4302 tablet_bootstrap.cc:492] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a: Bootstrap complete.
I20240502 14:15:48.235008 4302 ts_tablet_manager.cc:1367] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.002s
I20240502 14:15:48.236081 4302 raft_consensus.cc:348] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:48.236454 4302 raft_consensus.cc:729] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:15:48.237804 4431 tablet_copy_client.cc:308] T c0d152eda1384b6792246b2f10d667da P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.200:46097
I20240502 14:15:48.237273 4302 consensus_queue.cc:260] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:48.238879 29594 tablet_copy_service.cc:140] P 4092902b3e674857a0147f1b4f2cdb86: Received BeginTabletCopySession request for tablet c0d152eda1384b6792246b2f10d667da from peer 89af7f423cf44717a08d6b142f0ed49d ({username='slave'} at 127.19.94.193:43933)
I20240502 14:15:48.239076 29594 tablet_copy_service.cc:161] P 4092902b3e674857a0147f1b4f2cdb86: Beginning new tablet copy session on tablet c0d152eda1384b6792246b2f10d667da from peer 89af7f423cf44717a08d6b142f0ed49d at {username='slave'} at 127.19.94.193:43933: session id = 89af7f423cf44717a08d6b142f0ed49d-c0d152eda1384b6792246b2f10d667da
I20240502 14:15:48.245898 29594 tablet_copy_source_session.cc:205] T c0d152eda1384b6792246b2f10d667da P 4092902b3e674857a0147f1b4f2cdb86: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:48.247236 4302 ts_tablet_manager.cc:1398] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.012s user 0.002s sys 0.001s
I20240502 14:15:48.253259 29463 tablet_copy_service.cc:342] P 68a3c2b17ed94e06aba66ed4178132ab: Request end of tablet copy session ba187e38a2754d60a12de9ac4856430a-01e6bc3689db46e2ad7d0484a29af287 received from {username='slave'} at 127.19.94.194:57813
I20240502 14:15:48.253419 29463 tablet_copy_service.cc:433] P 68a3c2b17ed94e06aba66ed4178132ab: ending tablet copy session ba187e38a2754d60a12de9ac4856430a-01e6bc3689db46e2ad7d0484a29af287 on tablet 01e6bc3689db46e2ad7d0484a29af287 with peer ba187e38a2754d60a12de9ac4856430a
I20240502 14:15:48.254205 4431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0d152eda1384b6792246b2f10d667da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:48.262280 4431 tablet_copy_client.cc:783] T c0d152eda1384b6792246b2f10d667da P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:48.262740 4431 tablet_copy_client.cc:647] T c0d152eda1384b6792246b2f10d667da P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:48.264645 4431 tablet_copy_client.cc:523] T c0d152eda1384b6792246b2f10d667da P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:48.270092 4431 tablet_bootstrap.cc:492] T c0d152eda1384b6792246b2f10d667da P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:15:48.278635 4431 tablet.cc:1722] T c0d152eda1384b6792246b2f10d667da P 89af7f423cf44717a08d6b142f0ed49d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:48.279167 4431 tablet.cc:1620] T c0d152eda1384b6792246b2f10d667da P 89af7f423cf44717a08d6b142f0ed49d: MemRowSet was empty: no flush needed.
I20240502 14:15:48.279273 4431 tablet_metadata.cc:623] Not flushing: waiting for 1 pins to be released.
I20240502 14:15:48.280325 4431 tablet.cc:1722] T c0d152eda1384b6792246b2f10d667da P 89af7f423cf44717a08d6b142f0ed49d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:48.281232 4431 tablet.cc:1620] T c0d152eda1384b6792246b2f10d667da P 89af7f423cf44717a08d6b142f0ed49d: MemRowSet was empty: no flush needed.
I20240502 14:15:48.281334 4431 tablet_metadata.cc:623] Not flushing: waiting for 1 pins to be released.
I20240502 14:15:48.282498 4431 tablet_bootstrap.cc:492] T c0d152eda1384b6792246b2f10d667da P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:48.282929 4431 tablet_bootstrap.cc:492] T c0d152eda1384b6792246b2f10d667da P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap complete.
I20240502 14:15:48.287823 4431 ts_tablet_manager.cc:1367] T c0d152eda1384b6792246b2f10d667da P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.010s
I20240502 14:15:48.288652 4431 raft_consensus.cc:348] T c0d152eda1384b6792246b2f10d667da P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:48.289036 4431 raft_consensus.cc:729] T c0d152eda1384b6792246b2f10d667da P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:15:48.289284 4431 consensus_queue.cc:260] T c0d152eda1384b6792246b2f10d667da P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:48.289942 4431 ts_tablet_manager.cc:1398] T c0d152eda1384b6792246b2f10d667da P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:48.291107 29594 tablet_copy_service.cc:342] P 4092902b3e674857a0147f1b4f2cdb86: Request end of tablet copy session 89af7f423cf44717a08d6b142f0ed49d-c0d152eda1384b6792246b2f10d667da received from {username='slave'} at 127.19.94.193:43933
I20240502 14:15:48.291235 29594 tablet_copy_service.cc:433] P 4092902b3e674857a0147f1b4f2cdb86: ending tablet copy session 89af7f423cf44717a08d6b142f0ed49d-c0d152eda1384b6792246b2f10d667da on tablet c0d152eda1384b6792246b2f10d667da with peer 89af7f423cf44717a08d6b142f0ed49d
I20240502 14:15:48.302754 26689 consensus_queue.cc:237] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:48.306066 26815 raft_consensus.cc:1261] T 2108a04310804945b368bcb5f88170e2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:15:48.306514 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2108a04310804945b368bcb5f88170e2. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:48.306823 3585 consensus_queue.cc:1035] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:48.309240 26558 raft_consensus.cc:1261] T 2108a04310804945b368bcb5f88170e2 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:48.310472 4382 consensus_queue.cc:1035] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:48.314383 4382 raft_consensus.cc:2932] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:15:48.315481 26558 raft_consensus.cc:2932] T 2108a04310804945b368bcb5f88170e2 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:15:48.321198 26815 raft_consensus.cc:2932] T 2108a04310804945b368bcb5f88170e2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:15:48.326644 26279 catalog_manager.cc:5484] T 2108a04310804945b368bcb5f88170e2 P ba187e38a2754d60a12de9ac4856430a reported cstate change: config changed from index -1 to 4, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:15:48.531558 4487 ts_tablet_manager.cc:902] T 1fb1ef75829641a89c4cac539f431d80 P 1cb67158133e4648a6eceba32d4dbba3: Initiating tablet copy from peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:15:48.543001 4487 tablet_copy_client.cc:308] T 1fb1ef75829641a89c4cac539f431d80 P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.195:44325
I20240502 14:15:48.543912 26709 tablet_copy_service.cc:140] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Received BeginTabletCopySession request for tablet 1fb1ef75829641a89c4cac539f431d80 from peer 1cb67158133e4648a6eceba32d4dbba3 ({username='slave'} at 127.19.94.197:44731)
I20240502 14:15:48.544075 26709 tablet_copy_service.cc:161] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Beginning new tablet copy session on tablet 1fb1ef75829641a89c4cac539f431d80 from peer 1cb67158133e4648a6eceba32d4dbba3 at {username='slave'} at 127.19.94.197:44731: session id = 1cb67158133e4648a6eceba32d4dbba3-1fb1ef75829641a89c4cac539f431d80
I20240502 14:15:48.549000 26709 tablet_copy_source_session.cc:205] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:48.551156 4487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fb1ef75829641a89c4cac539f431d80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:48.556855 4492 ts_tablet_manager.cc:902] T 929a85cbe95746b798a7510b3a1cc19e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Initiating tablet copy from peer 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:15:48.564095 4487 tablet_copy_client.cc:783] T 1fb1ef75829641a89c4cac539f431d80 P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:48.565557 4487 tablet_copy_client.cc:647] T 1fb1ef75829641a89c4cac539f431d80 P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:48.567817 4487 tablet_copy_client.cc:523] T 1fb1ef75829641a89c4cac539f431d80 P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:48.578212 4487 tablet_bootstrap.cc:492] T 1fb1ef75829641a89c4cac539f431d80 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:15:48.607738 4431 ts_tablet_manager.cc:902] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d: Initiating tablet copy from peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:15:48.652401 4431 tablet_copy_client.cc:308] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.198:43659
I20240502 14:15:48.653321 29329 tablet_copy_service.cc:140] P c1100e16b3b9495ca61d4f95e89b6a89: Received BeginTabletCopySession request for tablet d6c8fa23fffe491fbfcec03afdb2052a from peer 89af7f423cf44717a08d6b142f0ed49d ({username='slave'} at 127.19.94.193:58815)
I20240502 14:15:48.653501 29329 tablet_copy_service.cc:161] P c1100e16b3b9495ca61d4f95e89b6a89: Beginning new tablet copy session on tablet d6c8fa23fffe491fbfcec03afdb2052a from peer 89af7f423cf44717a08d6b142f0ed49d at {username='slave'} at 127.19.94.193:58815: session id = 89af7f423cf44717a08d6b142f0ed49d-d6c8fa23fffe491fbfcec03afdb2052a
I20240502 14:15:48.658852 29329 tablet_copy_source_session.cc:205] T d6c8fa23fffe491fbfcec03afdb2052a P c1100e16b3b9495ca61d4f95e89b6a89: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:48.663887 26555 raft_consensus.cc:1203] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:48.668030 4431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6c8fa23fffe491fbfcec03afdb2052a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:48.671010 4492 tablet_copy_client.cc:308] T 929a85cbe95746b798a7510b3a1cc19e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.197:38715
I20240502 14:15:48.672178 29196 tablet_copy_service.cc:140] P 1cb67158133e4648a6eceba32d4dbba3: Received BeginTabletCopySession request for tablet 929a85cbe95746b798a7510b3a1cc19e from peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 ({username='slave'} at 127.19.94.195:56139)
I20240502 14:15:48.672433 29196 tablet_copy_service.cc:161] P 1cb67158133e4648a6eceba32d4dbba3: Beginning new tablet copy session on tablet 929a85cbe95746b798a7510b3a1cc19e from peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 at {username='slave'} at 127.19.94.195:56139: session id = 69ab78b2a1b74d2bbf2cdeafa0a4af23-929a85cbe95746b798a7510b3a1cc19e
I20240502 14:15:48.675751 29196 tablet_copy_source_session.cc:205] T 929a85cbe95746b798a7510b3a1cc19e P 1cb67158133e4648a6eceba32d4dbba3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:48.677556 4492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 929a85cbe95746b798a7510b3a1cc19e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:48.684909 26427 raft_consensus.cc:1203] T c0d152eda1384b6792246b2f10d667da P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:15:48.685833 4492 tablet_copy_client.cc:783] T 929a85cbe95746b798a7510b3a1cc19e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:48.686111 4492 tablet_copy_client.cc:647] T 929a85cbe95746b798a7510b3a1cc19e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:48.693240 4492 tablet_copy_client.cc:523] T 929a85cbe95746b798a7510b3a1cc19e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:48.693367 4431 tablet_copy_client.cc:783] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:48.693614 4431 tablet_copy_client.cc:647] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:48.710011 4492 tablet_bootstrap.cc:492] T 929a85cbe95746b798a7510b3a1cc19e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:15:48.710099 4431 tablet_copy_client.cc:523] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:48.714690 4431 tablet_bootstrap.cc:492] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:15:48.723845 4492 tablet_bootstrap.cc:492] T 929a85cbe95746b798a7510b3a1cc19e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:48.724339 4492 tablet_bootstrap.cc:492] T 929a85cbe95746b798a7510b3a1cc19e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap complete.
I20240502 14:15:48.724898 4492 ts_tablet_manager.cc:1367] T 929a85cbe95746b798a7510b3a1cc19e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.001s
I20240502 14:15:48.725804 4492 raft_consensus.cc:348] T 929a85cbe95746b798a7510b3a1cc19e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:48.726056 4492 raft_consensus.cc:729] T 929a85cbe95746b798a7510b3a1cc19e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:15:48.726408 4492 consensus_queue.cc:260] T 929a85cbe95746b798a7510b3a1cc19e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:48.727216 4492 ts_tablet_manager.cc:1398] T 929a85cbe95746b798a7510b3a1cc19e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:48.733012 29196 tablet_copy_service.cc:342] P 1cb67158133e4648a6eceba32d4dbba3: Request end of tablet copy session 69ab78b2a1b74d2bbf2cdeafa0a4af23-929a85cbe95746b798a7510b3a1cc19e received from {username='slave'} at 127.19.94.195:56139
I20240502 14:15:48.733178 29196 tablet_copy_service.cc:433] P 1cb67158133e4648a6eceba32d4dbba3: ending tablet copy session 69ab78b2a1b74d2bbf2cdeafa0a4af23-929a85cbe95746b798a7510b3a1cc19e on tablet 929a85cbe95746b798a7510b3a1cc19e with peer 69ab78b2a1b74d2bbf2cdeafa0a4af23
I20240502 14:15:48.737387 4431 tablet_bootstrap.cc:492] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:48.738577 4431 tablet_bootstrap.cc:492] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap complete.
I20240502 14:15:48.739594 4431 ts_tablet_manager.cc:1367] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.025s user 0.010s sys 0.000s
I20240502 14:15:48.740438 4431 raft_consensus.cc:348] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:48.740701 4431 raft_consensus.cc:729] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:15:48.741233 4487 tablet_bootstrap.cc:492] T 1fb1ef75829641a89c4cac539f431d80 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:48.741693 4487 tablet_bootstrap.cc:492] T 1fb1ef75829641a89c4cac539f431d80 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap complete.
I20240502 14:15:48.742254 4487 ts_tablet_manager.cc:1367] T 1fb1ef75829641a89c4cac539f431d80 P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.165s user 0.047s sys 0.005s
I20240502 14:15:48.740958 4431 consensus_queue.cc:260] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:48.743227 4431 ts_tablet_manager.cc:1398] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:15:48.744051 4487 raft_consensus.cc:348] T 1fb1ef75829641a89c4cac539f431d80 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:48.744289 4487 raft_consensus.cc:729] T 1fb1ef75829641a89c4cac539f431d80 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:15:48.744743 4487 consensus_queue.cc:260] T 1fb1ef75829641a89c4cac539f431d80 P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:48.745386 4487 ts_tablet_manager.cc:1398] T 1fb1ef75829641a89c4cac539f431d80 P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:15:48.746305 29329 tablet_copy_service.cc:342] P c1100e16b3b9495ca61d4f95e89b6a89: Request end of tablet copy session 89af7f423cf44717a08d6b142f0ed49d-d6c8fa23fffe491fbfcec03afdb2052a received from {username='slave'} at 127.19.94.193:58815
I20240502 14:15:48.746423 29329 tablet_copy_service.cc:433] P c1100e16b3b9495ca61d4f95e89b6a89: ending tablet copy session 89af7f423cf44717a08d6b142f0ed49d-d6c8fa23fffe491fbfcec03afdb2052a on tablet d6c8fa23fffe491fbfcec03afdb2052a with peer 89af7f423cf44717a08d6b142f0ed49d
I20240502 14:15:48.747829 26709 tablet_copy_service.cc:342] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Request end of tablet copy session 1cb67158133e4648a6eceba32d4dbba3-1fb1ef75829641a89c4cac539f431d80 received from {username='slave'} at 127.19.94.197:44731
I20240502 14:15:48.747951 26709 tablet_copy_service.cc:433] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: ending tablet copy session 1cb67158133e4648a6eceba32d4dbba3-1fb1ef75829641a89c4cac539f431d80 on tablet 1fb1ef75829641a89c4cac539f431d80 with peer 1cb67158133e4648a6eceba32d4dbba3
I20240502 14:15:48.779803 29155 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:48.800536 4526 ts_tablet_manager.cc:902] T 2108a04310804945b368bcb5f88170e2 P 68a3c2b17ed94e06aba66ed4178132ab: Initiating tablet copy from peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:15:48.872944 4526 tablet_copy_client.cc:308] T 2108a04310804945b368bcb5f88170e2 P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.195:44325
I20240502 14:15:48.876209 26709 tablet_copy_service.cc:140] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Received BeginTabletCopySession request for tablet 2108a04310804945b368bcb5f88170e2 from peer 68a3c2b17ed94e06aba66ed4178132ab ({username='slave'} at 127.19.94.199:51755)
I20240502 14:15:48.876613 26709 tablet_copy_service.cc:161] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Beginning new tablet copy session on tablet 2108a04310804945b368bcb5f88170e2 from peer 68a3c2b17ed94e06aba66ed4178132ab at {username='slave'} at 127.19.94.199:51755: session id = 68a3c2b17ed94e06aba66ed4178132ab-2108a04310804945b368bcb5f88170e2
I20240502 14:15:48.880036 26709 tablet_copy_source_session.cc:205] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:48.895653 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:48.905598 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:48.905882 4526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2108a04310804945b368bcb5f88170e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:48.914444 29176 raft_consensus.cc:1203] T 1fb1ef75829641a89c4cac539f431d80 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:15:48.917649 4526 tablet_copy_client.cc:783] T 2108a04310804945b368bcb5f88170e2 P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:48.918042 4526 tablet_copy_client.cc:647] T 2108a04310804945b368bcb5f88170e2 P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:48.927811 26686 raft_consensus.cc:1203] T 929a85cbe95746b798a7510b3a1cc19e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:48.930105 4526 tablet_copy_client.cc:523] T 2108a04310804945b368bcb5f88170e2 P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:48.932650 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:48.936429 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:48.938136 4526 tablet_bootstrap.cc:492] T 2108a04310804945b368bcb5f88170e2 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
W20240502 14:15:48.983824 26362 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.197:38715, user_credentials={real_user=slave}} blocked reactor thread for 106475us
I20240502 14:15:48.996012 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:48.998457 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:49.036952 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:49.111089 4526 tablet_bootstrap.cc:492] T 2108a04310804945b368bcb5f88170e2 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:49.116449 4526 tablet_bootstrap.cc:492] T 2108a04310804945b368bcb5f88170e2 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap complete.
I20240502 14:15:49.116941 4526 ts_tablet_manager.cc:1367] T 2108a04310804945b368bcb5f88170e2 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.179s user 0.053s sys 0.001s
I20240502 14:15:49.118037 4526 raft_consensus.cc:348] T 2108a04310804945b368bcb5f88170e2 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:49.125963 4526 raft_consensus.cc:729] T 2108a04310804945b368bcb5f88170e2 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:15:49.126372 4526 consensus_queue.cc:260] T 2108a04310804945b368bcb5f88170e2 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:49.136782 4526 ts_tablet_manager.cc:1398] T 2108a04310804945b368bcb5f88170e2 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20240502 14:15:49.137764 26709 tablet_copy_service.cc:342] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Request end of tablet copy session 68a3c2b17ed94e06aba66ed4178132ab-2108a04310804945b368bcb5f88170e2 received from {username='slave'} at 127.19.94.199:51755
I20240502 14:15:49.137939 26709 tablet_copy_service.cc:433] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: ending tablet copy session 68a3c2b17ed94e06aba66ed4178132ab-2108a04310804945b368bcb5f88170e2 on tablet 2108a04310804945b368bcb5f88170e2 with peer 68a3c2b17ed94e06aba66ed4178132ab
I20240502 14:15:49.153398 26427 raft_consensus.cc:1203] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:49.215349 29442 raft_consensus.cc:1203] T 2108a04310804945b368bcb5f88170e2 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:15:49.634763 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:49.683573 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:49.737365 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:49.738643 29155 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:49.744061 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:49.774474 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:49.785693 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:49.793184 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:49.990700 26689 consensus_queue.cc:237] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:15:49.992697 26425 raft_consensus.cc:1261] T 1fb1ef75829641a89c4cac539f431d80 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:49.992694 29174 raft_consensus.cc:1261] T 1fb1ef75829641a89c4cac539f431d80 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:49.993230 4455 consensus_queue.cc:1035] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:49.993480 4382 consensus_queue.cc:1035] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:49.996353 4382 raft_consensus.cc:2932] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:49.998056 26425 raft_consensus.cc:2932] T 1fb1ef75829641a89c4cac539f431d80 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:49.998387 29174 raft_consensus.cc:2932] T 1fb1ef75829641a89c4cac539f431d80 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:50.003516 26281 catalog_manager.cc:5484] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: config changed from index 4 to 5, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:15:50.020705 26798 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fb1ef75829641a89c4cac539f431d80 with delete_type TABLET_DATA_TOMBSTONED (TS e4cfb8fc5a474177af6848a6efa13a27 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:56010
I20240502 14:15:50.029142 4638 tablet_replica.cc:330] T 1fb1ef75829641a89c4cac539f431d80 P e4cfb8fc5a474177af6848a6efa13a27: stopping tablet replica
I20240502 14:15:50.029433 4638 raft_consensus.cc:2229] T 1fb1ef75829641a89c4cac539f431d80 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:50.029733 4638 raft_consensus.cc:2258] T 1fb1ef75829641a89c4cac539f431d80 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:50.030582 4638 ts_tablet_manager.cc:1875] T 1fb1ef75829641a89c4cac539f431d80 P e4cfb8fc5a474177af6848a6efa13a27: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:50.035339 4638 ts_tablet_manager.cc:1888] T 1fb1ef75829641a89c4cac539f431d80 P e4cfb8fc5a474177af6848a6efa13a27: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:15:50.035545 4638 log.cc:1196] T 1fb1ef75829641a89c4cac539f431d80 P e4cfb8fc5a474177af6848a6efa13a27: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/wals/1fb1ef75829641a89c4cac539f431d80
I20240502 14:15:50.036947 26272 catalog_manager.cc:4830] TS e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): tablet 1fb1ef75829641a89c4cac539f431d80 (table rebalance_test_table_13 [id=835ba73992f74008a1a10f4c18f2db38]) successfully deleted
I20240502 14:15:50.434955 29155 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:50.458603 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:50.468056 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:50.489415 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:50.586306 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:50.608124 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 14:15:50.643744 26754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.194:36025, user_credentials={real_user=slave}} blocked reactor thread for 70897.3us
I20240502 14:15:50.672253 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:50.683511 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:50.799368 29574 consensus_queue.cc:237] T c0d152eda1384b6792246b2f10d667da P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:50.801952 26427 raft_consensus.cc:1261] T c0d152eda1384b6792246b2f10d667da P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:50.801975 26555 raft_consensus.cc:1261] T c0d152eda1384b6792246b2f10d667da P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:50.802546 4032 consensus_queue.cc:1035] T c0d152eda1384b6792246b2f10d667da P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:50.802845 4545 consensus_queue.cc:1035] T c0d152eda1384b6792246b2f10d667da P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:50.805948 4032 raft_consensus.cc:2932] T c0d152eda1384b6792246b2f10d667da P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:50.806481 26555 raft_consensus.cc:2932] T c0d152eda1384b6792246b2f10d667da P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:50.806430 26427 raft_consensus.cc:2932] T c0d152eda1384b6792246b2f10d667da P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:50.813819 26278 catalog_manager.cc:5484] T c0d152eda1384b6792246b2f10d667da P ba187e38a2754d60a12de9ac4856430a reported cstate change: config changed from index 4 to 5, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:50.829914 29421 tablet_service.cc:1514] Processing DeleteTablet for tablet c0d152eda1384b6792246b2f10d667da with delete_type TABLET_DATA_TOMBSTONED (TS 68a3c2b17ed94e06aba66ed4178132ab not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:40652
I20240502 14:15:50.841264 4734 tablet_replica.cc:330] T c0d152eda1384b6792246b2f10d667da P 68a3c2b17ed94e06aba66ed4178132ab: stopping tablet replica
I20240502 14:15:50.841632 4734 raft_consensus.cc:2229] T c0d152eda1384b6792246b2f10d667da P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:50.842083 4734 raft_consensus.cc:2258] T c0d152eda1384b6792246b2f10d667da P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:50.844883 4734 ts_tablet_manager.cc:1875] T c0d152eda1384b6792246b2f10d667da P 68a3c2b17ed94e06aba66ed4178132ab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:50.852893 4734 ts_tablet_manager.cc:1888] T c0d152eda1384b6792246b2f10d667da P 68a3c2b17ed94e06aba66ed4178132ab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:15:50.853107 4734 log.cc:1196] T c0d152eda1384b6792246b2f10d667da P 68a3c2b17ed94e06aba66ed4178132ab: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal/wals/c0d152eda1384b6792246b2f10d667da
I20240502 14:15:50.853955 26272 catalog_manager.cc:4830] TS 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): tablet c0d152eda1384b6792246b2f10d667da (table rebalancer_dynamic_table [id=f9780b3608294a1380a60ae3416d3829]) successfully deleted
I20240502 14:15:51.163071 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:51.167325 29155 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:51.224316 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:51.228900 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:51.291069 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:51.291534 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:51.318852 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:51.383562 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:51.533178 29442 consensus_queue.cc:237] T 01e6bc3689db46e2ad7d0484a29af287 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:51.536551 26555 raft_consensus.cc:1261] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:51.536551 29174 raft_consensus.cc:1261] T 01e6bc3689db46e2ad7d0484a29af287 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:51.537174 4514 consensus_queue.cc:1035] T 01e6bc3689db46e2ad7d0484a29af287 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:51.537505 4513 consensus_queue.cc:1035] T 01e6bc3689db46e2ad7d0484a29af287 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:51.540328 4643 raft_consensus.cc:2932] T 01e6bc3689db46e2ad7d0484a29af287 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:51.545544 26555 raft_consensus.cc:2932] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:51.549525 29174 raft_consensus.cc:2932] T 01e6bc3689db46e2ad7d0484a29af287 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:51.554477 26279 catalog_manager.cc:5484] T 01e6bc3689db46e2ad7d0484a29af287 P 68a3c2b17ed94e06aba66ed4178132ab reported cstate change: config changed from index 2 to 3, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) evicted. New cstate: current_term: 1 leader_uuid: "68a3c2b17ed94e06aba66ed4178132ab" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:51.559321 29554 tablet_service.cc:1514] Processing DeleteTablet for tablet 01e6bc3689db46e2ad7d0484a29af287 with delete_type TABLET_DATA_TOMBSTONED (TS 4092902b3e674857a0147f1b4f2cdb86 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:36398
I20240502 14:15:51.559929 4818 tablet_replica.cc:330] T 01e6bc3689db46e2ad7d0484a29af287 P 4092902b3e674857a0147f1b4f2cdb86: stopping tablet replica
I20240502 14:15:51.560179 4818 raft_consensus.cc:2229] T 01e6bc3689db46e2ad7d0484a29af287 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:51.560431 4818 raft_consensus.cc:2258] T 01e6bc3689db46e2ad7d0484a29af287 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:51.561151 4818 ts_tablet_manager.cc:1875] T 01e6bc3689db46e2ad7d0484a29af287 P 4092902b3e674857a0147f1b4f2cdb86: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:51.569679 4818 ts_tablet_manager.cc:1888] T 01e6bc3689db46e2ad7d0484a29af287 P 4092902b3e674857a0147f1b4f2cdb86: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:15:51.569813 4818 log.cc:1196] T 01e6bc3689db46e2ad7d0484a29af287 P 4092902b3e674857a0147f1b4f2cdb86: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/wal/wals/01e6bc3689db46e2ad7d0484a29af287
I20240502 14:15:51.570609 26272 catalog_manager.cc:4830] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097): tablet 01e6bc3689db46e2ad7d0484a29af287 (table rebalancer_extra_table_6 [id=fe880af4989f416d82ae711576173990]) successfully deleted
I20240502 14:15:51.979898 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:51.985450 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:52.010318 29155 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:15:52.024283 26625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.196:32843, user_credentials={real_user=slave}} blocked reactor thread for 81392.2us
I20240502 14:15:52.027441 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:52.032917 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:52.065901 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:52.108383 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:52.156708 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:52.315161 29438 consensus_queue.cc:237] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:52.317639 26686 raft_consensus.cc:1261] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:52.317632 29307 raft_consensus.cc:1261] T e93acdb9427943938f4c12ae2bf2c594 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:52.318190 4514 consensus_queue.cc:1035] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:52.318471 4643 consensus_queue.cc:1035] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:52.321614 4909 raft_consensus.cc:2932] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:52.322131 26686 raft_consensus.cc:2932] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:52.322288 29307 raft_consensus.cc:2932] T e93acdb9427943938f4c12ae2bf2c594 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:52.331437 26279 catalog_manager.cc:5484] T e93acdb9427943938f4c12ae2bf2c594 P c1100e16b3b9495ca61d4f95e89b6a89 reported cstate change: config changed from index 2 to 3, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) evicted. New cstate: current_term: 1 leader_uuid: "68a3c2b17ed94e06aba66ed4178132ab" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:52.341693 29155 tablet_service.cc:1514] Processing DeleteTablet for tablet e93acdb9427943938f4c12ae2bf2c594 with delete_type TABLET_DATA_TOMBSTONED (TS 1cb67158133e4648a6eceba32d4dbba3 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:55202
I20240502 14:15:52.342293 4914 tablet_replica.cc:330] T e93acdb9427943938f4c12ae2bf2c594 P 1cb67158133e4648a6eceba32d4dbba3: stopping tablet replica
I20240502 14:15:52.342510 4914 raft_consensus.cc:2229] T e93acdb9427943938f4c12ae2bf2c594 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:52.342736 4914 raft_consensus.cc:2258] T e93acdb9427943938f4c12ae2bf2c594 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:52.343319 4914 ts_tablet_manager.cc:1875] T e93acdb9427943938f4c12ae2bf2c594 P 1cb67158133e4648a6eceba32d4dbba3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:52.350407 4914 ts_tablet_manager.cc:1888] T e93acdb9427943938f4c12ae2bf2c594 P 1cb67158133e4648a6eceba32d4dbba3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:15:52.350630 4914 log.cc:1196] T e93acdb9427943938f4c12ae2bf2c594 P 1cb67158133e4648a6eceba32d4dbba3: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal/wals/e93acdb9427943938f4c12ae2bf2c594
I20240502 14:15:52.351487 26271 catalog_manager.cc:4830] TS 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): tablet e93acdb9427943938f4c12ae2bf2c594 (table rebalancer_extra_table_0 [id=7727fea7ae174c949e336e765c50e53d]) successfully deleted
I20240502 14:15:52.689221 29155 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:15:52.705389 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:52.705658 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:52.755416 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:52.783133 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:52.825212 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:52.887794 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:15:52.917848 26755 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.200:46097, user_credentials={real_user=slave}} blocked reactor thread for 52065.6us
I20240502 14:15:52.936300 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:53.041426 26686 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "220adc4ff7dc4a3b957e479da22fbd3d"
dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52916
I20240502 14:15:53.041762 26686 raft_consensus.cc:595] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:15:53.109622 4974 raft_consensus.cc:979] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23: : Instructing follower 1cb67158133e4648a6eceba32d4dbba3 to start an election
I20240502 14:15:53.109882 4973 raft_consensus.cc:1067] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Signalling peer 1cb67158133e4648a6eceba32d4dbba3 to start an election
I20240502 14:15:53.111735 29176 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "220adc4ff7dc4a3b957e479da22fbd3d"
dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
from {username='slave'} at 127.19.94.195:56139
I20240502 14:15:53.112380 29176 raft_consensus.cc:482] T 220adc4ff7dc4a3b957e479da22fbd3d P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:53.112579 29176 raft_consensus.cc:3037] T 220adc4ff7dc4a3b957e479da22fbd3d P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:53.115691 29176 raft_consensus.cc:504] T 220adc4ff7dc4a3b957e479da22fbd3d P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:53.116775 29176 leader_election.cc:288] T 220adc4ff7dc4a3b957e479da22fbd3d P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 2 election: Requested vote from peers 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:15:53.117215 26427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220adc4ff7dc4a3b957e479da22fbd3d" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:15:53.117424 26427 raft_consensus.cc:3037] T 220adc4ff7dc4a3b957e479da22fbd3d P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:53.119931 26427 raft_consensus.cc:2454] T 220adc4ff7dc4a3b957e479da22fbd3d P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 2.
I20240502 14:15:53.120924 29573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220adc4ff7dc4a3b957e479da22fbd3d" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "4092902b3e674857a0147f1b4f2cdb86"
I20240502 14:15:53.121217 29573 raft_consensus.cc:3037] T 220adc4ff7dc4a3b957e479da22fbd3d P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:53.121817 26686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220adc4ff7dc4a3b957e479da22fbd3d" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:15:53.122085 26686 raft_consensus.cc:3032] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:53.122161 26686 raft_consensus.cc:729] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Running, Role: LEADER
I20240502 14:15:53.122524 26686 consensus_queue.cc:260] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:53.123126 26686 raft_consensus.cc:3037] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:53.125397 26686 raft_consensus.cc:2454] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 2.
I20240502 14:15:53.127099 29112 leader_election.cc:302] T 220adc4ff7dc4a3b957e479da22fbd3d P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 69ab78b2a1b74d2bbf2cdeafa0a4af23, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:15:53.127450 4686 raft_consensus.cc:2781] T 220adc4ff7dc4a3b957e479da22fbd3d P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:53.127570 4686 raft_consensus.cc:686] T 220adc4ff7dc4a3b957e479da22fbd3d P 1cb67158133e4648a6eceba32d4dbba3 [term 2 LEADER]: Becoming Leader. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Running, Role: LEADER
I20240502 14:15:53.127843 4686 consensus_queue.cc:237] T 220adc4ff7dc4a3b957e479da22fbd3d P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:15:53.130344 29573 raft_consensus.cc:2454] T 220adc4ff7dc4a3b957e479da22fbd3d P 4092902b3e674857a0147f1b4f2cdb86 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 2.
I20240502 14:15:53.146759 26283 catalog_manager.cc:5484] T 220adc4ff7dc4a3b957e479da22fbd3d P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: term changed from 1 to 2, leader changed from 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) to 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197). New cstate: current_term: 2 leader_uuid: "1cb67158133e4648a6eceba32d4dbba3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:15:53.423945 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:53.454557 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:53.491931 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:53.513279 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:53.569826 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:53.641110 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:53.653635 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:53.659091 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:53.687237 26686 raft_consensus.cc:1261] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 2 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:15:53.687979 4686 consensus_queue.cc:1035] T 220adc4ff7dc4a3b957e479da22fbd3d P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:53.694875 26425 raft_consensus.cc:1261] T 220adc4ff7dc4a3b957e479da22fbd3d P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:15:53.695498 4686 consensus_queue.cc:1035] T 220adc4ff7dc4a3b957e479da22fbd3d P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:53.742760 29573 raft_consensus.cc:1261] T 220adc4ff7dc4a3b957e479da22fbd3d P 4092902b3e674857a0147f1b4f2cdb86 [term 2 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:15:53.743492 4686 consensus_queue.cc:1035] T 220adc4ff7dc4a3b957e479da22fbd3d P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:54.279757 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:54.305625 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:54.321723 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:54.355897 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:54.357143 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:54.391090 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:54.430608 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:54.434540 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:54.661227 29442 consensus_queue.cc:237] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:15:54.663452 26556 raft_consensus.cc:1261] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:54.663477 29307 raft_consensus.cc:1261] T 457516f59dee4436b474af63819fad4e P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:54.664093 4909 consensus_queue.cc:1035] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:54.664439 4998 consensus_queue.cc:1035] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:54.668321 5144 raft_consensus.cc:2932] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:54.669004 26556 raft_consensus.cc:2932] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:54.673245 29307 raft_consensus.cc:2932] T 457516f59dee4436b474af63819fad4e P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:54.678505 26279 catalog_manager.cc:5484] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab reported cstate change: config changed from index 2 to 3, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) evicted. New cstate: current_term: 1 leader_uuid: "68a3c2b17ed94e06aba66ed4178132ab" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:15:54.686342 29155 tablet_service.cc:1514] Processing DeleteTablet for tablet 457516f59dee4436b474af63819fad4e with delete_type TABLET_DATA_TOMBSTONED (TS 1cb67158133e4648a6eceba32d4dbba3 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:55202
I20240502 14:15:54.688069 5150 tablet_replica.cc:330] T 457516f59dee4436b474af63819fad4e P 1cb67158133e4648a6eceba32d4dbba3: stopping tablet replica
I20240502 14:15:54.688283 5150 raft_consensus.cc:2229] T 457516f59dee4436b474af63819fad4e P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:54.688503 5150 raft_consensus.cc:2258] T 457516f59dee4436b474af63819fad4e P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:54.689086 5150 ts_tablet_manager.cc:1875] T 457516f59dee4436b474af63819fad4e P 1cb67158133e4648a6eceba32d4dbba3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:54.693698 5150 ts_tablet_manager.cc:1888] T 457516f59dee4436b474af63819fad4e P 1cb67158133e4648a6eceba32d4dbba3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:15:54.693807 5150 log.cc:1196] T 457516f59dee4436b474af63819fad4e P 1cb67158133e4648a6eceba32d4dbba3: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal/wals/457516f59dee4436b474af63819fad4e
I20240502 14:15:54.694936 26271 catalog_manager.cc:4830] TS 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): tablet 457516f59dee4436b474af63819fad4e (table rebalancer_extra_table_32 [id=eb9f13fa5bdc4cc3b3c074bc196dfb8c]) successfully deleted
I20240502 14:15:54.961052 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:54.977458 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:55.001003 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:55.022768 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:55.046764 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:55.048453 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:55.092836 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:55.218673 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:55.357825 26425 consensus_queue.cc:237] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:55.367646 26818 raft_consensus.cc:1261] T 822e90c09531419e9b32af0438fceb57 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:55.367750 29442 raft_consensus.cc:1261] T 822e90c09531419e9b32af0438fceb57 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:55.368747 3816 consensus_queue.cc:1035] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:55.369378 3816 consensus_queue.cc:1035] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:55.384315 3816 raft_consensus.cc:2932] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:55.385013 29442 raft_consensus.cc:2932] T 822e90c09531419e9b32af0438fceb57 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:55.395728 26281 catalog_manager.cc:5484] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:55.400763 26818 raft_consensus.cc:2932] T 822e90c09531419e9b32af0438fceb57 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:55.422958 29287 tablet_service.cc:1514] Processing DeleteTablet for tablet 822e90c09531419e9b32af0438fceb57 with delete_type TABLET_DATA_TOMBSTONED (TS c1100e16b3b9495ca61d4f95e89b6a89 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:59160
I20240502 14:15:55.423638 5239 tablet_replica.cc:330] T 822e90c09531419e9b32af0438fceb57 P c1100e16b3b9495ca61d4f95e89b6a89: stopping tablet replica
I20240502 14:15:55.423858 5239 raft_consensus.cc:2229] T 822e90c09531419e9b32af0438fceb57 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:55.424099 5239 raft_consensus.cc:2258] T 822e90c09531419e9b32af0438fceb57 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:55.424865 5239 ts_tablet_manager.cc:1875] T 822e90c09531419e9b32af0438fceb57 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:55.431535 5239 ts_tablet_manager.cc:1888] T 822e90c09531419e9b32af0438fceb57 P c1100e16b3b9495ca61d4f95e89b6a89: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:15:55.431799 5239 log.cc:1196] T 822e90c09531419e9b32af0438fceb57 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal/wals/822e90c09531419e9b32af0438fceb57
I20240502 14:15:55.432689 26271 catalog_manager.cc:4830] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): tablet 822e90c09531419e9b32af0438fceb57 (table rebalancer_extra_table_16 [id=15e0d2ef8fa244be97447fb977d23693]) successfully deleted
I20240502 14:15:55.700773 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:55.720702 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:55.752686 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:55.785388 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:55.807654 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:55.879575 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:55.886098 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:55.973598 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:56.082685 29307 consensus_queue.cc:237] T d6c8fa23fffe491fbfcec03afdb2052a P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:15:56.085002 26686 raft_consensus.cc:1261] T d6c8fa23fffe491fbfcec03afdb2052a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer c1100e16b3b9495ca61d4f95e89b6a89: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:56.085098 26426 raft_consensus.cc:1261] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer c1100e16b3b9495ca61d4f95e89b6a89: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:56.085579 4883 consensus_queue.cc:1035] T d6c8fa23fffe491fbfcec03afdb2052a P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:56.085867 5030 consensus_queue.cc:1035] T d6c8fa23fffe491fbfcec03afdb2052a P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:56.098073 4883 raft_consensus.cc:2932] T d6c8fa23fffe491fbfcec03afdb2052a P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:56.101207 26686 raft_consensus.cc:2932] T d6c8fa23fffe491fbfcec03afdb2052a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:56.107693 26426 raft_consensus.cc:2932] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:56.109326 26276 catalog_manager.cc:5484] T d6c8fa23fffe491fbfcec03afdb2052a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: config changed from index 2 to 3, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) evicted. New cstate: current_term: 1 leader_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:15:56.139124 29554 tablet_service.cc:1514] Processing DeleteTablet for tablet d6c8fa23fffe491fbfcec03afdb2052a with delete_type TABLET_DATA_TOMBSTONED (TS 4092902b3e674857a0147f1b4f2cdb86 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:36398
I20240502 14:15:56.142980 5333 tablet_replica.cc:330] T d6c8fa23fffe491fbfcec03afdb2052a P 4092902b3e674857a0147f1b4f2cdb86: stopping tablet replica
I20240502 14:15:56.143242 5333 raft_consensus.cc:2229] T d6c8fa23fffe491fbfcec03afdb2052a P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:56.143497 5333 raft_consensus.cc:2258] T d6c8fa23fffe491fbfcec03afdb2052a P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:56.145309 5333 ts_tablet_manager.cc:1875] T d6c8fa23fffe491fbfcec03afdb2052a P 4092902b3e674857a0147f1b4f2cdb86: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:56.154732 5333 ts_tablet_manager.cc:1888] T d6c8fa23fffe491fbfcec03afdb2052a P 4092902b3e674857a0147f1b4f2cdb86: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:15:56.155951 5333 log.cc:1196] T d6c8fa23fffe491fbfcec03afdb2052a P 4092902b3e674857a0147f1b4f2cdb86: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/wal/wals/d6c8fa23fffe491fbfcec03afdb2052a
I20240502 14:15:56.158006 26272 catalog_manager.cc:4830] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097): tablet d6c8fa23fffe491fbfcec03afdb2052a (table rebalancer_extra_table_30 [id=50fafe33ec534b21a087e7ea3eab5a91]) successfully deleted
I20240502 14:15:56.504649 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:56.505720 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:56.522207 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:56.541707 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:56.553233 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:56.599488 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:56.632825 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:56.680440 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:56.814819 26426 consensus_queue.cc:237] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:15:56.817174 26555 raft_consensus.cc:1261] T 6814613d36a74353a99673cf16aa9821 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:56.817245 29309 raft_consensus.cc:1261] T 6814613d36a74353a99673cf16aa9821 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:56.817821 3816 consensus_queue.cc:1035] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:56.818169 5210 consensus_queue.cc:1035] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:56.822224 3816 raft_consensus.cc:2932] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:15:56.828406 29309 raft_consensus.cc:2932] T 6814613d36a74353a99673cf16aa9821 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:15:56.828605 26555 raft_consensus.cc:2932] T 6814613d36a74353a99673cf16aa9821 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:15:56.838910 26279 catalog_manager.cc:5484] T 6814613d36a74353a99673cf16aa9821 P c1100e16b3b9495ca61d4f95e89b6a89 reported cstate change: config changed from index 4 to 5, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:15:56.875613 26798 tablet_service.cc:1514] Processing DeleteTablet for tablet 6814613d36a74353a99673cf16aa9821 with delete_type TABLET_DATA_TOMBSTONED (TS e4cfb8fc5a474177af6848a6efa13a27 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:56010
I20240502 14:15:56.879128 5427 tablet_replica.cc:330] T 6814613d36a74353a99673cf16aa9821 P e4cfb8fc5a474177af6848a6efa13a27: stopping tablet replica
I20240502 14:15:56.879464 5427 raft_consensus.cc:2229] T 6814613d36a74353a99673cf16aa9821 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:56.880411 5427 raft_consensus.cc:2258] T 6814613d36a74353a99673cf16aa9821 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:56.881278 5427 ts_tablet_manager.cc:1875] T 6814613d36a74353a99673cf16aa9821 P e4cfb8fc5a474177af6848a6efa13a27: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:56.889225 5427 ts_tablet_manager.cc:1888] T 6814613d36a74353a99673cf16aa9821 P e4cfb8fc5a474177af6848a6efa13a27: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:15:56.889473 5427 log.cc:1196] T 6814613d36a74353a99673cf16aa9821 P e4cfb8fc5a474177af6848a6efa13a27: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/wals/6814613d36a74353a99673cf16aa9821
I20240502 14:15:56.890266 26272 catalog_manager.cc:4830] TS e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): tablet 6814613d36a74353a99673cf16aa9821 (table rebalance_test_table_13 [id=835ba73992f74008a1a10f4c18f2db38]) successfully deleted
I20240502 14:15:57.319744 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:57.352538 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:57.397902 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:57.399137 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240502 14:15:57.416211 29511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 85376.1us
I20240502 14:15:57.498255 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:57.514086 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:57.563174 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:57.565552 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:57.752629 26686 consensus_queue.cc:237] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:15:57.755883 26818 raft_consensus.cc:1261] T c41522c60f3942aa9f28f514dcad4e49 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:57.755873 29438 raft_consensus.cc:1261] T c41522c60f3942aa9f28f514dcad4e49 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:57.756690 5391 consensus_queue.cc:1035] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:57.757261 5391 consensus_queue.cc:1035] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:57.783021 5509 raft_consensus.cc:2932] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:15:57.790004 26283 catalog_manager.cc:5484] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: config changed from index 4 to 5, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) evicted. New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:15:57.802062 26818 raft_consensus.cc:2932] T c41522c60f3942aa9f28f514dcad4e49 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:15:57.803717 29439 raft_consensus.cc:2932] T c41522c60f3942aa9f28f514dcad4e49 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:15:57.823163 26406 tablet_service.cc:1514] Processing DeleteTablet for tablet c41522c60f3942aa9f28f514dcad4e49 with delete_type TABLET_DATA_TOMBSTONED (TS 89af7f423cf44717a08d6b142f0ed49d not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:37422
I20240502 14:15:57.823894 5519 tablet_replica.cc:330] T c41522c60f3942aa9f28f514dcad4e49 P 89af7f423cf44717a08d6b142f0ed49d: stopping tablet replica
I20240502 14:15:57.824680 5519 raft_consensus.cc:2229] T c41522c60f3942aa9f28f514dcad4e49 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:57.825018 5519 raft_consensus.cc:2258] T c41522c60f3942aa9f28f514dcad4e49 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:57.825706 5519 ts_tablet_manager.cc:1875] T c41522c60f3942aa9f28f514dcad4e49 P 89af7f423cf44717a08d6b142f0ed49d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:57.832963 5519 ts_tablet_manager.cc:1888] T c41522c60f3942aa9f28f514dcad4e49 P 89af7f423cf44717a08d6b142f0ed49d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:15:57.833096 5519 log.cc:1196] T c41522c60f3942aa9f28f514dcad4e49 P 89af7f423cf44717a08d6b142f0ed49d: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/c41522c60f3942aa9f28f514dcad4e49
I20240502 14:15:57.833986 26271 catalog_manager.cc:4830] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): tablet c41522c60f3942aa9f28f514dcad4e49 (table rebalance_test_table_13 [id=835ba73992f74008a1a10f4c18f2db38]) successfully deleted
I20240502 14:15:58.165295 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:58.213800 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:58.227742 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:58.254895 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:58.283560 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:58.289983 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:58.300258 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:58.337600 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:58.559382 29175 consensus_queue.cc:237] T f6561a18519245f9a99690259c9c295e P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:15:58.561772 26686 raft_consensus.cc:1261] T f6561a18519245f9a99690259c9c295e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:58.561801 26815 raft_consensus.cc:1261] T f6561a18519245f9a99690259c9c295e P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:58.562325 5299 consensus_queue.cc:1035] T f6561a18519245f9a99690259c9c295e P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:58.562623 4400 consensus_queue.cc:1035] T f6561a18519245f9a99690259c9c295e P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:58.565868 4400 raft_consensus.cc:2932] T f6561a18519245f9a99690259c9c295e P 1cb67158133e4648a6eceba32d4dbba3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:58.568312 26686 raft_consensus.cc:2932] T f6561a18519245f9a99690259c9c295e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:58.575992 26281 catalog_manager.cc:5484] T f6561a18519245f9a99690259c9c295e P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: config changed from index 2 to 3, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) evicted. New cstate: current_term: 1 leader_uuid: "1cb67158133e4648a6eceba32d4dbba3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:58.597978 26815 raft_consensus.cc:2932] T f6561a18519245f9a99690259c9c295e P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:15:58.618494 29554 tablet_service.cc:1514] Processing DeleteTablet for tablet f6561a18519245f9a99690259c9c295e with delete_type TABLET_DATA_TOMBSTONED (TS 4092902b3e674857a0147f1b4f2cdb86 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:36398
I20240502 14:15:58.633481 5598 tablet_replica.cc:330] T f6561a18519245f9a99690259c9c295e P 4092902b3e674857a0147f1b4f2cdb86: stopping tablet replica
I20240502 14:15:58.633744 5598 raft_consensus.cc:2229] T f6561a18519245f9a99690259c9c295e P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:58.634044 5598 raft_consensus.cc:2258] T f6561a18519245f9a99690259c9c295e P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:58.634711 5598 ts_tablet_manager.cc:1875] T f6561a18519245f9a99690259c9c295e P 4092902b3e674857a0147f1b4f2cdb86: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:58.645854 5598 ts_tablet_manager.cc:1888] T f6561a18519245f9a99690259c9c295e P 4092902b3e674857a0147f1b4f2cdb86: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:15:58.649698 5598 log.cc:1196] T f6561a18519245f9a99690259c9c295e P 4092902b3e674857a0147f1b4f2cdb86: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/wal/wals/f6561a18519245f9a99690259c9c295e
I20240502 14:15:58.655885 26272 catalog_manager.cc:4830] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097): tablet f6561a18519245f9a99690259c9c295e (table rebalancer_extra_table_26 [id=88ec6ba556824ae1ac8b611f69608326]) successfully deleted
I20240502 14:15:59.093204 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:59.095383 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:59.095991 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:59.097072 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:59.144768 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:59.206247 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:59.248800 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:59.286652 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 14:15:59.384727 26624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.193:37985, user_credentials={real_user=slave}} blocked reactor thread for 49804.5us
I20240502 14:15:59.464710 29439 consensus_queue.cc:237] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:15:59.468207 26818 raft_consensus.cc:1261] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:59.468567 26427 raft_consensus.cc:1261] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:59.468901 5693 consensus_queue.cc:1035] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:59.469202 5369 consensus_queue.cc:1035] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:59.472965 5693 raft_consensus.cc:2932] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:59.477063 26818 raft_consensus.cc:2932] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:59.477382 26427 raft_consensus.cc:2932] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:59.484184 26278 catalog_manager.cc:5484] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab reported cstate change: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New cstate: current_term: 1 leader_uuid: "68a3c2b17ed94e06aba66ed4178132ab" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:15:59.499784 29287 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cea6495041642f58e1eb5d26fc10df5 with delete_type TABLET_DATA_TOMBSTONED (TS c1100e16b3b9495ca61d4f95e89b6a89 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:59160
I20240502 14:15:59.512575 5705 tablet_replica.cc:330] T 4cea6495041642f58e1eb5d26fc10df5 P c1100e16b3b9495ca61d4f95e89b6a89: stopping tablet replica
I20240502 14:15:59.512982 5705 raft_consensus.cc:2229] T 4cea6495041642f58e1eb5d26fc10df5 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:59.513304 5705 raft_consensus.cc:2258] T 4cea6495041642f58e1eb5d26fc10df5 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:59.514166 5705 ts_tablet_manager.cc:1875] T 4cea6495041642f58e1eb5d26fc10df5 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:59.522991 5705 ts_tablet_manager.cc:1888] T 4cea6495041642f58e1eb5d26fc10df5 P c1100e16b3b9495ca61d4f95e89b6a89: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:15:59.523203 5705 log.cc:1196] T 4cea6495041642f58e1eb5d26fc10df5 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal/wals/4cea6495041642f58e1eb5d26fc10df5
I20240502 14:15:59.528517 26271 catalog_manager.cc:4830] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): tablet 4cea6495041642f58e1eb5d26fc10df5 (table rebalancer_extra_table_10 [id=77b7f84f16494751ad1f1e50fa6a8db0]) successfully deleted
I20240502 14:15:59.842983 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:59.843811 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:59.865375 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:59.890354 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:15:59.916879 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:15:59.933683 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:59.945681 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:59.985495 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:00.161449 26688 consensus_queue.cc:237] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:16:00.172449 26556 raft_consensus.cc:1261] T 870100caa8bf4ca1b49337ba374b1f83 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:00.173270 5391 consensus_queue.cc:1035] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:00.179010 29439 raft_consensus.cc:1261] T 870100caa8bf4ca1b49337ba374b1f83 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:00.179711 5391 consensus_queue.cc:1035] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:00.189168 5391 raft_consensus.cc:2932] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:16:00.193185 29439 raft_consensus.cc:2932] T 870100caa8bf4ca1b49337ba374b1f83 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:16:00.199349 26279 catalog_manager.cc:5484] T 870100caa8bf4ca1b49337ba374b1f83 P 68a3c2b17ed94e06aba66ed4178132ab reported cstate change: config changed from index 2 to 3, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) evicted. New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:16:00.201516 26556 raft_consensus.cc:2932] T 870100caa8bf4ca1b49337ba374b1f83 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:16:00.228801 26406 tablet_service.cc:1514] Processing DeleteTablet for tablet 870100caa8bf4ca1b49337ba374b1f83 with delete_type TABLET_DATA_TOMBSTONED (TS 89af7f423cf44717a08d6b142f0ed49d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:37422
I20240502 14:16:00.232651 5788 tablet_replica.cc:330] T 870100caa8bf4ca1b49337ba374b1f83 P 89af7f423cf44717a08d6b142f0ed49d: stopping tablet replica
I20240502 14:16:00.232962 5788 raft_consensus.cc:2229] T 870100caa8bf4ca1b49337ba374b1f83 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:00.233245 5788 raft_consensus.cc:2258] T 870100caa8bf4ca1b49337ba374b1f83 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:00.234041 5788 ts_tablet_manager.cc:1875] T 870100caa8bf4ca1b49337ba374b1f83 P 89af7f423cf44717a08d6b142f0ed49d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:00.241024 5788 ts_tablet_manager.cc:1888] T 870100caa8bf4ca1b49337ba374b1f83 P 89af7f423cf44717a08d6b142f0ed49d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:16:00.241161 5788 log.cc:1196] T 870100caa8bf4ca1b49337ba374b1f83 P 89af7f423cf44717a08d6b142f0ed49d: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/870100caa8bf4ca1b49337ba374b1f83
I20240502 14:16:00.242091 26271 catalog_manager.cc:4830] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): tablet 870100caa8bf4ca1b49337ba374b1f83 (table rebalancer_extra_table_40 [id=ede5d7f4445942ccba7beb577ec03516]) successfully deleted
I20240502 14:16:00.688282 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:00.724674 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:00.728297 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:00.738106 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:16:00.762605 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:00.777698 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 14:16:00.829917 29244 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.197:38715, user_credentials={real_user=slave}} blocked reactor thread for 63373.2us
I20240502 14:16:00.837715 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:00.839646 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:00.983973 26688 consensus_queue.cc:237] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:16:00.985960 29438 raft_consensus.cc:1261] T 2108a04310804945b368bcb5f88170e2 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:00.986143 26815 raft_consensus.cc:1261] T 2108a04310804945b368bcb5f88170e2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:00.986492 5391 consensus_queue.cc:1035] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:00.986805 5855 consensus_queue.cc:1035] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:00.989480 5855 raft_consensus.cc:2932] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:00.991925 29438 raft_consensus.cc:2932] T 2108a04310804945b368bcb5f88170e2 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:01.000623 26278 catalog_manager.cc:5484] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: config changed from index 4 to 5, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) evicted. New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:01.008322 26815 raft_consensus.cc:2932] T 2108a04310804945b368bcb5f88170e2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:01.011559 26538 tablet_service.cc:1514] Processing DeleteTablet for tablet 2108a04310804945b368bcb5f88170e2 with delete_type TABLET_DATA_TOMBSTONED (TS ba187e38a2754d60a12de9ac4856430a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51696
I20240502 14:16:01.035707 5881 tablet_replica.cc:330] T 2108a04310804945b368bcb5f88170e2 P ba187e38a2754d60a12de9ac4856430a: stopping tablet replica
I20240502 14:16:01.035948 5881 raft_consensus.cc:2229] T 2108a04310804945b368bcb5f88170e2 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:01.036223 5881 raft_consensus.cc:2258] T 2108a04310804945b368bcb5f88170e2 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:01.042579 5881 ts_tablet_manager.cc:1875] T 2108a04310804945b368bcb5f88170e2 P ba187e38a2754d60a12de9ac4856430a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:01.057502 5881 ts_tablet_manager.cc:1888] T 2108a04310804945b368bcb5f88170e2 P ba187e38a2754d60a12de9ac4856430a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:16:01.057647 5881 log.cc:1196] T 2108a04310804945b368bcb5f88170e2 P ba187e38a2754d60a12de9ac4856430a: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/2108a04310804945b368bcb5f88170e2
I20240502 14:16:01.058482 26271 catalog_manager.cc:4830] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): tablet 2108a04310804945b368bcb5f88170e2 (table rebalance_test_table_9 [id=3c3eeaa25da94395a642aa569b652348]) successfully deleted
I20240502 14:16:01.432580 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:01.471324 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:01.488809 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:01.489820 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:01.574216 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:01.634055 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:01.645110 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 14:16:01.714051 29378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.197:38715, user_credentials={real_user=slave}} blocked reactor thread for 75077.3us
I20240502 14:16:01.733561 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:16:01.870707 29309 consensus_queue.cc:237] T 3f6d55d89e55425183da73dbb3f4175f P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:01.873493 26425 raft_consensus.cc:1261] T 3f6d55d89e55425183da73dbb3f4175f P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer c1100e16b3b9495ca61d4f95e89b6a89: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:01.873528 26556 raft_consensus.cc:1261] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer c1100e16b3b9495ca61d4f95e89b6a89: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:01.874145 5030 consensus_queue.cc:1035] T 3f6d55d89e55425183da73dbb3f4175f P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:01.874503 5854 consensus_queue.cc:1035] T 3f6d55d89e55425183da73dbb3f4175f P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:01.877672 5854 raft_consensus.cc:2932] T 3f6d55d89e55425183da73dbb3f4175f P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:01.878736 26556 raft_consensus.cc:2932] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:01.881004 26425 raft_consensus.cc:2932] T 3f6d55d89e55425183da73dbb3f4175f P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:01.883705 26277 catalog_manager.cc:5484] T 3f6d55d89e55425183da73dbb3f4175f P c1100e16b3b9495ca61d4f95e89b6a89 reported cstate change: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New cstate: current_term: 1 leader_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:01.910831 29421 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f6d55d89e55425183da73dbb3f4175f with delete_type TABLET_DATA_TOMBSTONED (TS 68a3c2b17ed94e06aba66ed4178132ab not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:40652
I20240502 14:16:01.911646 5985 tablet_replica.cc:330] T 3f6d55d89e55425183da73dbb3f4175f P 68a3c2b17ed94e06aba66ed4178132ab: stopping tablet replica
I20240502 14:16:01.912256 5985 raft_consensus.cc:2229] T 3f6d55d89e55425183da73dbb3f4175f P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:01.912523 5985 raft_consensus.cc:2258] T 3f6d55d89e55425183da73dbb3f4175f P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:01.913240 5985 ts_tablet_manager.cc:1875] T 3f6d55d89e55425183da73dbb3f4175f P 68a3c2b17ed94e06aba66ed4178132ab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:01.919792 5985 ts_tablet_manager.cc:1888] T 3f6d55d89e55425183da73dbb3f4175f P 68a3c2b17ed94e06aba66ed4178132ab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:16:01.919932 5985 log.cc:1196] T 3f6d55d89e55425183da73dbb3f4175f P 68a3c2b17ed94e06aba66ed4178132ab: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal/wals/3f6d55d89e55425183da73dbb3f4175f
I20240502 14:16:01.920789 26272 catalog_manager.cc:4830] TS 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): tablet 3f6d55d89e55425183da73dbb3f4175f (table rebalancer_extra_table_24 [id=8ce70558d80e412a89355b4917fc1a4e]) successfully deleted
W20240502 14:16:02.288908 29112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.196:32843, user_credentials={real_user=slave}} blocked reactor thread for 43897.7us
I20240502 14:16:02.465888 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:02.478238 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:02.535496 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:02.588166 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:02.638381 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:16:02.753172 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:02.755853 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:02.812261 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:02.946655 29438 consensus_queue.cc:237] T c32ea761f2834d758529890e5b496f88 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:16:02.949059 29307 raft_consensus.cc:1261] T c32ea761f2834d758529890e5b496f88 P c1100e16b3b9495ca61d4f95e89b6a89 [term 2 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:16:02.949100 26815 raft_consensus.cc:1261] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:16:02.949677 5955 consensus_queue.cc:1035] T c32ea761f2834d758529890e5b496f88 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:02.952646 5955 consensus_queue.cc:1035] T c32ea761f2834d758529890e5b496f88 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:02.953527 5954 raft_consensus.cc:2932] T c32ea761f2834d758529890e5b496f88 P 68a3c2b17ed94e06aba66ed4178132ab [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:02.954036 26815 raft_consensus.cc:2932] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:02.960577 26282 catalog_manager.cc:5484] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: config changed from index 2 to 4, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) evicted. New cstate: current_term: 2 leader_uuid: "68a3c2b17ed94e06aba66ed4178132ab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:02.977402 29307 raft_consensus.cc:2932] T c32ea761f2834d758529890e5b496f88 P c1100e16b3b9495ca61d4f95e89b6a89 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:02.979326 29155 tablet_service.cc:1514] Processing DeleteTablet for tablet c32ea761f2834d758529890e5b496f88 with delete_type TABLET_DATA_TOMBSTONED (TS 1cb67158133e4648a6eceba32d4dbba3 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55202
I20240502 14:16:02.980023 6083 tablet_replica.cc:330] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3: stopping tablet replica
I20240502 14:16:02.980257 6083 raft_consensus.cc:2229] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:02.980468 6083 raft_consensus.cc:2258] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:02.981076 6083 ts_tablet_manager.cc:1875] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:02.987622 6083 ts_tablet_manager.cc:1888] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240502 14:16:02.987882 6083 log.cc:1196] T c32ea761f2834d758529890e5b496f88 P 1cb67158133e4648a6eceba32d4dbba3: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal/wals/c32ea761f2834d758529890e5b496f88
I20240502 14:16:02.989092 26271 catalog_manager.cc:4830] TS 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): tablet c32ea761f2834d758529890e5b496f88 (table rebalancer_extra_table_12 [id=48a10accbab847d6b456b86bacb19a13]) successfully deleted
I20240502 14:16:03.025571 26425 consensus_queue.cc:237] T aedd72fd3aeb48d68f26a042069cebab P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
W20240502 14:16:03.029083 26363 consensus_peers.cc:485] T aedd72fd3aeb48d68f26a042069cebab P 89af7f423cf44717a08d6b142f0ed49d -> Peer 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097): Couldn't send request to peer 4092902b3e674857a0147f1b4f2cdb86. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aedd72fd3aeb48d68f26a042069cebab. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:03.030625 26815 raft_consensus.cc:1261] T aedd72fd3aeb48d68f26a042069cebab P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:03.031363 26687 raft_consensus.cc:1261] T aedd72fd3aeb48d68f26a042069cebab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:03.031313 5412 consensus_queue.cc:1035] T aedd72fd3aeb48d68f26a042069cebab P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:03.034044 5412 consensus_queue.cc:1035] T aedd72fd3aeb48d68f26a042069cebab P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:03.048219 5412 raft_consensus.cc:2932] T aedd72fd3aeb48d68f26a042069cebab P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:16:03.054849 26686 raft_consensus.cc:2932] T aedd72fd3aeb48d68f26a042069cebab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:16:03.055320 26277 catalog_manager.cc:5484] T aedd72fd3aeb48d68f26a042069cebab P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: config changed from index -1 to 4, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:16:03.055665 26815 raft_consensus.cc:2932] T aedd72fd3aeb48d68f26a042069cebab P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:16:03.184190 26558 consensus_queue.cc:237] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
W20240502 14:16:03.202837 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 946239523c0646f1835d7a85b4e7d117. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:03.204208 26815 raft_consensus.cc:1261] T 946239523c0646f1835d7a85b4e7d117 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:03.204962 5906 consensus_queue.cc:1035] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:03.206818 26687 raft_consensus.cc:1261] T 946239523c0646f1835d7a85b4e7d117 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:03.208019 4997 consensus_queue.cc:1035] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:03.220491 4997 raft_consensus.cc:2932] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:03.221513 26818 raft_consensus.cc:2932] T 946239523c0646f1835d7a85b4e7d117 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:03.226511 26686 raft_consensus.cc:2932] T 946239523c0646f1835d7a85b4e7d117 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:03.232017 26282 catalog_manager.cc:5484] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a reported cstate change: config changed from index -1 to 4, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:03.645867 6166 ts_tablet_manager.cc:902] T 946239523c0646f1835d7a85b4e7d117 P c1100e16b3b9495ca61d4f95e89b6a89: Initiating tablet copy from peer ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
W20240502 14:16:03.647289 29510 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.197:38715, user_credentials={real_user=slave}} blocked reactor thread for 53326.5us
I20240502 14:16:03.658541 6164 ts_tablet_manager.cc:902] T aedd72fd3aeb48d68f26a042069cebab P 4092902b3e674857a0147f1b4f2cdb86: Initiating tablet copy from peer 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:16:03.726557 6164 tablet_copy_client.cc:308] T aedd72fd3aeb48d68f26a042069cebab P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.193:37985
I20240502 14:16:03.727473 26447 tablet_copy_service.cc:140] P 89af7f423cf44717a08d6b142f0ed49d: Received BeginTabletCopySession request for tablet aedd72fd3aeb48d68f26a042069cebab from peer 4092902b3e674857a0147f1b4f2cdb86 ({username='slave'} at 127.19.94.200:60351)
I20240502 14:16:03.727658 26447 tablet_copy_service.cc:161] P 89af7f423cf44717a08d6b142f0ed49d: Beginning new tablet copy session on tablet aedd72fd3aeb48d68f26a042069cebab from peer 4092902b3e674857a0147f1b4f2cdb86 at {username='slave'} at 127.19.94.200:60351: session id = 4092902b3e674857a0147f1b4f2cdb86-aedd72fd3aeb48d68f26a042069cebab
I20240502 14:16:03.727941 6166 tablet_copy_client.cc:308] T 946239523c0646f1835d7a85b4e7d117 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.194:36025
I20240502 14:16:03.730650 26447 tablet_copy_source_session.cc:205] T aedd72fd3aeb48d68f26a042069cebab P 89af7f423cf44717a08d6b142f0ed49d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:03.745767 6164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aedd72fd3aeb48d68f26a042069cebab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:03.754741 6164 tablet_copy_client.cc:783] T aedd72fd3aeb48d68f26a042069cebab P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:03.755053 6164 tablet_copy_client.cc:647] T aedd72fd3aeb48d68f26a042069cebab P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Starting download of 1 WAL segments...
W20240502 14:16:03.755594 29244 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.197:38715, user_credentials={real_user=slave}} blocked reactor thread for 64458.1us
I20240502 14:16:03.756508 26578 tablet_copy_service.cc:140] P ba187e38a2754d60a12de9ac4856430a: Received BeginTabletCopySession request for tablet 946239523c0646f1835d7a85b4e7d117 from peer c1100e16b3b9495ca61d4f95e89b6a89 ({username='slave'} at 127.19.94.198:36021)
I20240502 14:16:03.756700 26578 tablet_copy_service.cc:161] P ba187e38a2754d60a12de9ac4856430a: Beginning new tablet copy session on tablet 946239523c0646f1835d7a85b4e7d117 from peer c1100e16b3b9495ca61d4f95e89b6a89 at {username='slave'} at 127.19.94.198:36021: session id = c1100e16b3b9495ca61d4f95e89b6a89-946239523c0646f1835d7a85b4e7d117
I20240502 14:16:03.760051 6164 tablet_copy_client.cc:523] T aedd72fd3aeb48d68f26a042069cebab P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:03.761624 26578 tablet_copy_source_session.cc:205] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:03.763971 6166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 946239523c0646f1835d7a85b4e7d117. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:03.775624 6166 tablet_copy_client.cc:783] T 946239523c0646f1835d7a85b4e7d117 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:03.777649 6166 tablet_copy_client.cc:647] T 946239523c0646f1835d7a85b4e7d117 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:03.781837 6166 tablet_copy_client.cc:523] T 946239523c0646f1835d7a85b4e7d117 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:03.783782 6164 tablet_bootstrap.cc:492] T aedd72fd3aeb48d68f26a042069cebab P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:16:03.793398 6166 tablet_bootstrap.cc:492] T 946239523c0646f1835d7a85b4e7d117 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:16:03.879864 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:03.945853 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:03.958813 6164 tablet_bootstrap.cc:492] T aedd72fd3aeb48d68f26a042069cebab P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:03.959297 6164 tablet_bootstrap.cc:492] T aedd72fd3aeb48d68f26a042069cebab P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap complete.
I20240502 14:16:03.962011 6164 ts_tablet_manager.cc:1367] T aedd72fd3aeb48d68f26a042069cebab P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.178s user 0.051s sys 0.007s
I20240502 14:16:03.962800 6164 raft_consensus.cc:348] T aedd72fd3aeb48d68f26a042069cebab P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:16:03.963182 6164 raft_consensus.cc:729] T aedd72fd3aeb48d68f26a042069cebab P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:16:03.963510 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:03.964715 6166 tablet_bootstrap.cc:492] T 946239523c0646f1835d7a85b4e7d117 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:03.965319 6166 tablet_bootstrap.cc:492] T 946239523c0646f1835d7a85b4e7d117 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap complete.
I20240502 14:16:03.965602 6166 ts_tablet_manager.cc:1367] T 946239523c0646f1835d7a85b4e7d117 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.172s user 0.054s sys 0.005s
I20240502 14:16:03.966297 6164 consensus_queue.cc:260] T aedd72fd3aeb48d68f26a042069cebab P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:16:03.967078 6164 ts_tablet_manager.cc:1398] T aedd72fd3aeb48d68f26a042069cebab P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 14:16:03.967828 6166 raft_consensus.cc:348] T 946239523c0646f1835d7a85b4e7d117 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:16:03.968065 6166 raft_consensus.cc:729] T 946239523c0646f1835d7a85b4e7d117 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:16:03.968309 6166 consensus_queue.cc:260] T 946239523c0646f1835d7a85b4e7d117 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:16:03.969098 6166 ts_tablet_manager.cc:1398] T 946239523c0646f1835d7a85b4e7d117 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:16:03.973114 26578 tablet_copy_service.cc:342] P ba187e38a2754d60a12de9ac4856430a: Request end of tablet copy session c1100e16b3b9495ca61d4f95e89b6a89-946239523c0646f1835d7a85b4e7d117 received from {username='slave'} at 127.19.94.198:36021
I20240502 14:16:03.973258 26578 tablet_copy_service.cc:433] P ba187e38a2754d60a12de9ac4856430a: ending tablet copy session c1100e16b3b9495ca61d4f95e89b6a89-946239523c0646f1835d7a85b4e7d117 on tablet 946239523c0646f1835d7a85b4e7d117 with peer c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:16:03.974840 26447 tablet_copy_service.cc:342] P 89af7f423cf44717a08d6b142f0ed49d: Request end of tablet copy session 4092902b3e674857a0147f1b4f2cdb86-aedd72fd3aeb48d68f26a042069cebab received from {username='slave'} at 127.19.94.200:60351
I20240502 14:16:03.974992 26447 tablet_copy_service.cc:433] P 89af7f423cf44717a08d6b142f0ed49d: ending tablet copy session 4092902b3e674857a0147f1b4f2cdb86-aedd72fd3aeb48d68f26a042069cebab on tablet aedd72fd3aeb48d68f26a042069cebab with peer 4092902b3e674857a0147f1b4f2cdb86
I20240502 14:16:04.002197 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:04.003852 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:16:04.018483 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:04.018693 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:04.029387 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:04.064529 29307 raft_consensus.cc:1203] T 946239523c0646f1835d7a85b4e7d117 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:16:04.075242 29573 raft_consensus.cc:1203] T aedd72fd3aeb48d68f26a042069cebab P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240502 14:16:04.590147 26363 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.196:32843, user_credentials={real_user=slave}} blocked reactor thread for 46551.2us
I20240502 14:16:04.709769 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:04.736928 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:04.739455 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:04.796907 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:16:04.815703 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:04.841569 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:04.849915 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:04.907186 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:05.058804 26426 consensus_queue.cc:237] T aedd72fd3aeb48d68f26a042069cebab P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:16:05.061223 29573 raft_consensus.cc:1261] T aedd72fd3aeb48d68f26a042069cebab P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:05.061213 26688 raft_consensus.cc:1261] T aedd72fd3aeb48d68f26a042069cebab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:05.061921 6096 consensus_queue.cc:1035] T aedd72fd3aeb48d68f26a042069cebab P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:05.062292 6251 consensus_queue.cc:1035] T aedd72fd3aeb48d68f26a042069cebab P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:05.065608 6251 raft_consensus.cc:2932] T aedd72fd3aeb48d68f26a042069cebab P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:16:05.066433 26688 raft_consensus.cc:2932] T aedd72fd3aeb48d68f26a042069cebab P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:16:05.074462 26285 catalog_manager.cc:5484] T aedd72fd3aeb48d68f26a042069cebab P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: config changed from index 4 to 5, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:16:05.077853 29573 raft_consensus.cc:2932] T aedd72fd3aeb48d68f26a042069cebab P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:16:05.125838 26798 tablet_service.cc:1514] Processing DeleteTablet for tablet aedd72fd3aeb48d68f26a042069cebab with delete_type TABLET_DATA_TOMBSTONED (TS e4cfb8fc5a474177af6848a6efa13a27 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:56010
I20240502 14:16:05.126955 6304 tablet_replica.cc:330] T aedd72fd3aeb48d68f26a042069cebab P e4cfb8fc5a474177af6848a6efa13a27: stopping tablet replica
I20240502 14:16:05.128836 6304 raft_consensus.cc:2229] T aedd72fd3aeb48d68f26a042069cebab P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:05.129236 6304 raft_consensus.cc:2258] T aedd72fd3aeb48d68f26a042069cebab P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:05.130043 6304 ts_tablet_manager.cc:1875] T aedd72fd3aeb48d68f26a042069cebab P e4cfb8fc5a474177af6848a6efa13a27: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:05.134727 6304 ts_tablet_manager.cc:1888] T aedd72fd3aeb48d68f26a042069cebab P e4cfb8fc5a474177af6848a6efa13a27: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:16:05.134855 6304 log.cc:1196] T aedd72fd3aeb48d68f26a042069cebab P e4cfb8fc5a474177af6848a6efa13a27: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/wals/aedd72fd3aeb48d68f26a042069cebab
I20240502 14:16:05.135756 26272 catalog_manager.cc:4830] TS e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): tablet aedd72fd3aeb48d68f26a042069cebab (table rebalance_test_table_7 [id=998b08830b5f4b6fbb49d494f21739b9]) successfully deleted
I20240502 14:16:05.529356 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:05.530277 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:05.579977 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:16:05.586431 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:05.588640 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:05.599857 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:05.612661 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:05.669045 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:05.819411 29176 consensus_queue.cc:237] T 929a85cbe95746b798a7510b3a1cc19e P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:16:05.821892 26818 raft_consensus.cc:1261] T 929a85cbe95746b798a7510b3a1cc19e P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:05.821938 26686 raft_consensus.cc:1261] T 929a85cbe95746b798a7510b3a1cc19e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:05.822512 6105 consensus_queue.cc:1035] T 929a85cbe95746b798a7510b3a1cc19e P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:05.822968 6029 consensus_queue.cc:1035] T 929a85cbe95746b798a7510b3a1cc19e P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:05.826112 6029 raft_consensus.cc:2932] T 929a85cbe95746b798a7510b3a1cc19e P 1cb67158133e4648a6eceba32d4dbba3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:05.826661 26818 raft_consensus.cc:2932] T 929a85cbe95746b798a7510b3a1cc19e P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:05.827231 26686 raft_consensus.cc:2932] T 929a85cbe95746b798a7510b3a1cc19e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:05.834704 26280 catalog_manager.cc:5484] T 929a85cbe95746b798a7510b3a1cc19e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New cstate: current_term: 1 leader_uuid: "1cb67158133e4648a6eceba32d4dbba3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:05.842608 29287 tablet_service.cc:1514] Processing DeleteTablet for tablet 929a85cbe95746b798a7510b3a1cc19e with delete_type TABLET_DATA_TOMBSTONED (TS c1100e16b3b9495ca61d4f95e89b6a89 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:59160
I20240502 14:16:05.844303 6392 tablet_replica.cc:330] T 929a85cbe95746b798a7510b3a1cc19e P c1100e16b3b9495ca61d4f95e89b6a89: stopping tablet replica
I20240502 14:16:05.844657 6392 raft_consensus.cc:2229] T 929a85cbe95746b798a7510b3a1cc19e P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:05.845001 6392 raft_consensus.cc:2258] T 929a85cbe95746b798a7510b3a1cc19e P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:05.848148 6392 ts_tablet_manager.cc:1875] T 929a85cbe95746b798a7510b3a1cc19e P c1100e16b3b9495ca61d4f95e89b6a89: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:05.858043 6392 ts_tablet_manager.cc:1888] T 929a85cbe95746b798a7510b3a1cc19e P c1100e16b3b9495ca61d4f95e89b6a89: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:16:05.858229 6392 log.cc:1196] T 929a85cbe95746b798a7510b3a1cc19e P c1100e16b3b9495ca61d4f95e89b6a89: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal/wals/929a85cbe95746b798a7510b3a1cc19e
I20240502 14:16:05.860061 26271 catalog_manager.cc:4830] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): tablet 929a85cbe95746b798a7510b3a1cc19e (table rebalancer_extra_table_18 [id=800c8a30773545b3a7670200a5873c0e]) successfully deleted
I20240502 14:16:06.258291 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:06.294564 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:06.299005 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:06.340278 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:06.352316 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:16:06.379418 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:06.460855 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:06.492336 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:06.649600 29176 consensus_queue.cc:237] T 220adc4ff7dc4a3b957e479da22fbd3d P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:16:06.651916 29573 raft_consensus.cc:1261] T 220adc4ff7dc4a3b957e479da22fbd3d P 4092902b3e674857a0147f1b4f2cdb86 [term 2 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:16:06.652521 26427 raft_consensus.cc:1261] T 220adc4ff7dc4a3b957e479da22fbd3d P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:16:06.652616 6338 consensus_queue.cc:1035] T 220adc4ff7dc4a3b957e479da22fbd3d P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:06.653188 6105 consensus_queue.cc:1035] T 220adc4ff7dc4a3b957e479da22fbd3d P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:06.656076 6105 raft_consensus.cc:2932] T 220adc4ff7dc4a3b957e479da22fbd3d P 1cb67158133e4648a6eceba32d4dbba3 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:16:06.656803 29573 raft_consensus.cc:2932] T 220adc4ff7dc4a3b957e479da22fbd3d P 4092902b3e674857a0147f1b4f2cdb86 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:16:06.657449 26427 raft_consensus.cc:2932] T 220adc4ff7dc4a3b957e479da22fbd3d P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:16:06.664364 26282 catalog_manager.cc:5484] T 220adc4ff7dc4a3b957e479da22fbd3d P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: config changed from index 2 to 4, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New cstate: current_term: 2 leader_uuid: "1cb67158133e4648a6eceba32d4dbba3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:16:06.688419 26669 tablet_service.cc:1514] Processing DeleteTablet for tablet 220adc4ff7dc4a3b957e479da22fbd3d with delete_type TABLET_DATA_TOMBSTONED (TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34150
I20240502 14:16:06.689043 6501 tablet_replica.cc:330] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23: stopping tablet replica
I20240502 14:16:06.689236 6501 raft_consensus.cc:2229] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:06.689401 6501 raft_consensus.cc:2258] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:06.689970 6501 ts_tablet_manager.cc:1875] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:06.695225 6501 ts_tablet_manager.cc:1888] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240502 14:16:06.695327 6501 log.cc:1196] T 220adc4ff7dc4a3b957e479da22fbd3d P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/wals/220adc4ff7dc4a3b957e479da22fbd3d
I20240502 14:16:06.696023 26272 catalog_manager.cc:4830] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): tablet 220adc4ff7dc4a3b957e479da22fbd3d (table rebalancer_extra_table_46 [id=cf2fa69bdc0c4ba4af966938f6e97bd7]) successfully deleted
I20240502 14:16:06.908123 29307 consensus_queue.cc:237] T 2e59791e95cc452ea30d1ed8bb6178ae P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:06.910791 26558 raft_consensus.cc:1261] T 2e59791e95cc452ea30d1ed8bb6178ae P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer c1100e16b3b9495ca61d4f95e89b6a89: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:06.911382 26687 raft_consensus.cc:1261] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer c1100e16b3b9495ca61d4f95e89b6a89: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:06.911459 6568 consensus_queue.cc:1035] T 2e59791e95cc452ea30d1ed8bb6178ae P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:16:06.911934 29244 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P c1100e16b3b9495ca61d4f95e89b6a89 -> Peer 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): Couldn't send request to peer 89af7f423cf44717a08d6b142f0ed49d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e59791e95cc452ea30d1ed8bb6178ae. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:06.911978 5854 consensus_queue.cc:1035] T 2e59791e95cc452ea30d1ed8bb6178ae P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:06.915539 5854 raft_consensus.cc:2932] T 2e59791e95cc452ea30d1ed8bb6178ae P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:06.925326 26558 raft_consensus.cc:2932] T 2e59791e95cc452ea30d1ed8bb6178ae P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:06.932366 26687 raft_consensus.cc:2932] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:06.934077 26280 catalog_manager.cc:5484] T 2e59791e95cc452ea30d1ed8bb6178ae P ba187e38a2754d60a12de9ac4856430a reported cstate change: config changed from index -1 to 2, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New cstate: current_term: 1 leader_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:06.979295 29573 consensus_queue.cc:237] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:16:06.986227 29440 raft_consensus.cc:1261] T 83dd34749ff8496483365b30a86db71f P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:06.988371 5304 consensus_queue.cc:1035] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:06.988945 29174 raft_consensus.cc:1261] T 83dd34749ff8496483365b30a86db71f P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:06.989518 5304 consensus_queue.cc:1035] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:16:06.990192 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): Couldn't send request to peer e4cfb8fc5a474177af6848a6efa13a27. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 83dd34749ff8496483365b30a86db71f. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:07.000973 6589 raft_consensus.cc:2932] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:07.001952 29174 raft_consensus.cc:2932] T 83dd34749ff8496483365b30a86db71f P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:07.009109 26278 catalog_manager.cc:5484] T 83dd34749ff8496483365b30a86db71f P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: config changed from index -1 to 2, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:07.011971 29438 raft_consensus.cc:2932] T 83dd34749ff8496483365b30a86db71f P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:07.120311 29307 consensus_queue.cc:237] T 33a3639c26bf4b35a6e15d6ef173751c P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:16:07.123878 29573 raft_consensus.cc:1261] T 33a3639c26bf4b35a6e15d6ef173751c P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Refusing update from remote peer c1100e16b3b9495ca61d4f95e89b6a89: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:07.124280 29438 raft_consensus.cc:1261] T 33a3639c26bf4b35a6e15d6ef173751c P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer c1100e16b3b9495ca61d4f95e89b6a89: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:16:07.124732 29244 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P c1100e16b3b9495ca61d4f95e89b6a89 -> Peer ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): Couldn't send request to peer ba187e38a2754d60a12de9ac4856430a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 33a3639c26bf4b35a6e15d6ef173751c. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:07.124869 6568 consensus_queue.cc:1035] T 33a3639c26bf4b35a6e15d6ef173751c P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:07.125435 5854 consensus_queue.cc:1035] T 33a3639c26bf4b35a6e15d6ef173751c P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:07.134537 6568 raft_consensus.cc:2932] T 33a3639c26bf4b35a6e15d6ef173751c P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:16:07.135244 29573 raft_consensus.cc:2932] T 33a3639c26bf4b35a6e15d6ef173751c P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:16:07.142495 26280 catalog_manager.cc:5484] T 33a3639c26bf4b35a6e15d6ef173751c P c1100e16b3b9495ca61d4f95e89b6a89 reported cstate change: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New cstate: current_term: 1 leader_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:16:07.143083 29440 raft_consensus.cc:2932] T 33a3639c26bf4b35a6e15d6ef173751c P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:16:07.238992 26815 consensus_queue.cc:237] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:16:07.243746 29307 raft_consensus.cc:1261] T e45de4b8415446a08e9550f911367624 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:07.246099 6368 consensus_queue.cc:1035] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:16:07.249210 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): Couldn't send request to peer 1cb67158133e4648a6eceba32d4dbba3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e45de4b8415446a08e9550f911367624. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:07.255580 26425 raft_consensus.cc:1261] T e45de4b8415446a08e9550f911367624 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:07.256100 6629 consensus_queue.cc:1035] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:07.271055 6368 raft_consensus.cc:2932] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:16:07.271821 29307 raft_consensus.cc:2932] T e45de4b8415446a08e9550f911367624 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:16:07.278343 26278 catalog_manager.cc:5484] T e45de4b8415446a08e9550f911367624 P c1100e16b3b9495ca61d4f95e89b6a89 reported cstate change: config changed from index -1 to 2, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:16:07.281428 26427 raft_consensus.cc:2932] T e45de4b8415446a08e9550f911367624 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:16:07.326865 26427 consensus_queue.cc:237] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:16:07.330049 29440 raft_consensus.cc:1261] T e1bdebd770244537a0119baa4e3f8091 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:07.330273 29176 raft_consensus.cc:1261] T e1bdebd770244537a0119baa4e3f8091 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:07.330665 6251 consensus_queue.cc:1035] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:16:07.330976 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e1bdebd770244537a0119baa4e3f8091. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:07.331248 6251 consensus_queue.cc:1035] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:07.335330 6649 raft_consensus.cc:2932] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:07.335939 29176 raft_consensus.cc:2932] T e1bdebd770244537a0119baa4e3f8091 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:07.342649 26278 catalog_manager.cc:5484] T e1bdebd770244537a0119baa4e3f8091 P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: config changed from index -1 to 2, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:07.349329 29440 raft_consensus.cc:2932] T e1bdebd770244537a0119baa4e3f8091 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:07.419318 6662 ts_tablet_manager.cc:902] T 2e59791e95cc452ea30d1ed8bb6178ae P 89af7f423cf44717a08d6b142f0ed49d: Initiating tablet copy from peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:16:07.429201 6662 tablet_copy_client.cc:308] T 2e59791e95cc452ea30d1ed8bb6178ae P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.198:43659
I20240502 14:16:07.430140 29329 tablet_copy_service.cc:140] P c1100e16b3b9495ca61d4f95e89b6a89: Received BeginTabletCopySession request for tablet 2e59791e95cc452ea30d1ed8bb6178ae from peer 89af7f423cf44717a08d6b142f0ed49d ({username='slave'} at 127.19.94.193:58815)
I20240502 14:16:07.430322 29329 tablet_copy_service.cc:161] P c1100e16b3b9495ca61d4f95e89b6a89: Beginning new tablet copy session on tablet 2e59791e95cc452ea30d1ed8bb6178ae from peer 89af7f423cf44717a08d6b142f0ed49d at {username='slave'} at 127.19.94.193:58815: session id = 89af7f423cf44717a08d6b142f0ed49d-2e59791e95cc452ea30d1ed8bb6178ae
I20240502 14:16:07.436298 29329 tablet_copy_source_session.cc:205] T 2e59791e95cc452ea30d1ed8bb6178ae P c1100e16b3b9495ca61d4f95e89b6a89: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:07.438215 6662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e59791e95cc452ea30d1ed8bb6178ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:07.447459 6662 tablet_copy_client.cc:783] T 2e59791e95cc452ea30d1ed8bb6178ae P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:07.447865 6662 tablet_copy_client.cc:647] T 2e59791e95cc452ea30d1ed8bb6178ae P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:07.450567 6662 tablet_copy_client.cc:523] T 2e59791e95cc452ea30d1ed8bb6178ae P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:07.451678 26558 consensus_queue.cc:237] T aa4c1d9bf17a48b1b657bb7d03409efb P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:16:07.457294 6662 tablet_bootstrap.cc:492] T 2e59791e95cc452ea30d1ed8bb6178ae P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:16:07.458218 29176 raft_consensus.cc:1261] T aa4c1d9bf17a48b1b657bb7d03409efb P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:07.458521 29573 raft_consensus.cc:1261] T aa4c1d9bf17a48b1b657bb7d03409efb P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:07.459558 6159 consensus_queue.cc:1035] T aa4c1d9bf17a48b1b657bb7d03409efb P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:16:07.459985 26494 consensus_peers.cc:485] T aa4c1d9bf17a48b1b657bb7d03409efb P ba187e38a2754d60a12de9ac4856430a -> Peer e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): Couldn't send request to peer e4cfb8fc5a474177af6848a6efa13a27. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aa4c1d9bf17a48b1b657bb7d03409efb. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:07.460191 6159 consensus_queue.cc:1035] T aa4c1d9bf17a48b1b657bb7d03409efb P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:07.477070 6159 raft_consensus.cc:2932] T aa4c1d9bf17a48b1b657bb7d03409efb P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:07.477923 29176 raft_consensus.cc:2932] T aa4c1d9bf17a48b1b657bb7d03409efb P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:07.484278 29573 raft_consensus.cc:2932] T aa4c1d9bf17a48b1b657bb7d03409efb P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:07.486054 6662 tablet_bootstrap.cc:492] T 2e59791e95cc452ea30d1ed8bb6178ae P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:07.486527 6662 tablet_bootstrap.cc:492] T 2e59791e95cc452ea30d1ed8bb6178ae P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap complete.
I20240502 14:16:07.486789 6662 ts_tablet_manager.cc:1367] T 2e59791e95cc452ea30d1ed8bb6178ae P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.001s
I20240502 14:16:07.487447 26282 catalog_manager.cc:5484] T aa4c1d9bf17a48b1b657bb7d03409efb P ba187e38a2754d60a12de9ac4856430a reported cstate change: config changed from index -1 to 2, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:07.487640 6662 raft_consensus.cc:348] T 2e59791e95cc452ea30d1ed8bb6178ae P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:07.487833 6662 raft_consensus.cc:729] T 2e59791e95cc452ea30d1ed8bb6178ae P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:16:07.488054 6662 consensus_queue.cc:260] T 2e59791e95cc452ea30d1ed8bb6178ae P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:07.488682 6662 ts_tablet_manager.cc:1398] T 2e59791e95cc452ea30d1ed8bb6178ae P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 14:16:07.489517 29329 tablet_copy_service.cc:342] P c1100e16b3b9495ca61d4f95e89b6a89: Request end of tablet copy session 89af7f423cf44717a08d6b142f0ed49d-2e59791e95cc452ea30d1ed8bb6178ae received from {username='slave'} at 127.19.94.193:58815
I20240502 14:16:07.489667 29329 tablet_copy_service.cc:433] P c1100e16b3b9495ca61d4f95e89b6a89: ending tablet copy session 89af7f423cf44717a08d6b142f0ed49d-2e59791e95cc452ea30d1ed8bb6178ae on tablet 2e59791e95cc452ea30d1ed8bb6178ae with peer 89af7f423cf44717a08d6b142f0ed49d
I20240502 14:16:07.538233 6690 ts_tablet_manager.cc:902] T 83dd34749ff8496483365b30a86db71f P e4cfb8fc5a474177af6848a6efa13a27: Initiating tablet copy from peer 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:16:07.540956 6690 tablet_copy_client.cc:308] T 83dd34749ff8496483365b30a86db71f P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.200:46097
I20240502 14:16:07.541797 29594 tablet_copy_service.cc:140] P 4092902b3e674857a0147f1b4f2cdb86: Received BeginTabletCopySession request for tablet 83dd34749ff8496483365b30a86db71f from peer e4cfb8fc5a474177af6848a6efa13a27 ({username='slave'} at 127.19.94.196:45221)
I20240502 14:16:07.541992 29594 tablet_copy_service.cc:161] P 4092902b3e674857a0147f1b4f2cdb86: Beginning new tablet copy session on tablet 83dd34749ff8496483365b30a86db71f from peer e4cfb8fc5a474177af6848a6efa13a27 at {username='slave'} at 127.19.94.196:45221: session id = e4cfb8fc5a474177af6848a6efa13a27-83dd34749ff8496483365b30a86db71f
I20240502 14:16:07.546705 29594 tablet_copy_source_session.cc:205] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:07.548897 6690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83dd34749ff8496483365b30a86db71f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:07.552305 26686 consensus_queue.cc:237] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:16:07.554605 6690 tablet_copy_client.cc:783] T 83dd34749ff8496483365b30a86db71f P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:07.554847 6690 tablet_copy_client.cc:647] T 83dd34749ff8496483365b30a86db71f P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:07.555291 29174 raft_consensus.cc:1261] T 451b303a32a3404d8d584203d6704b35 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:07.555820 26558 raft_consensus.cc:1261] T 451b303a32a3404d8d584203d6704b35 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:16:07.556278 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 451b303a32a3404d8d584203d6704b35. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:07.556485 6678 consensus_queue.cc:1035] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:07.557152 6678 consensus_queue.cc:1035] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:07.557934 6690 tablet_copy_client.cc:523] T 83dd34749ff8496483365b30a86db71f P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:07.562224 6690 tablet_bootstrap.cc:492] T 83dd34749ff8496483365b30a86db71f P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:16:07.567464 6312 raft_consensus.cc:2932] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:07.568513 29176 raft_consensus.cc:2932] T 451b303a32a3404d8d584203d6704b35 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:07.570739 6690 tablet_bootstrap.cc:492] T 83dd34749ff8496483365b30a86db71f P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:07.571132 6690 tablet_bootstrap.cc:492] T 83dd34749ff8496483365b30a86db71f P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap complete.
I20240502 14:16:07.573139 26558 raft_consensus.cc:2932] T 451b303a32a3404d8d584203d6704b35 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:07.575389 26280 catalog_manager.cc:5484] T 451b303a32a3404d8d584203d6704b35 P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: config changed from index -1 to 2, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:07.578348 6690 ts_tablet_manager.cc:1367] T 83dd34749ff8496483365b30a86db71f P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.006s
I20240502 14:16:07.579973 6690 raft_consensus.cc:348] T 83dd34749ff8496483365b30a86db71f P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:16:07.580240 6690 raft_consensus.cc:729] T 83dd34749ff8496483365b30a86db71f P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:16:07.580524 6690 consensus_queue.cc:260] T 83dd34749ff8496483365b30a86db71f P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:16:07.584369 6690 ts_tablet_manager.cc:1398] T 83dd34749ff8496483365b30a86db71f P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:16:07.586720 29594 tablet_copy_service.cc:342] P 4092902b3e674857a0147f1b4f2cdb86: Request end of tablet copy session e4cfb8fc5a474177af6848a6efa13a27-83dd34749ff8496483365b30a86db71f received from {username='slave'} at 127.19.94.196:45221
I20240502 14:16:07.586861 29594 tablet_copy_service.cc:433] P 4092902b3e674857a0147f1b4f2cdb86: ending tablet copy session e4cfb8fc5a474177af6848a6efa13a27-83dd34749ff8496483365b30a86db71f on tablet 83dd34749ff8496483365b30a86db71f with peer e4cfb8fc5a474177af6848a6efa13a27
I20240502 14:16:07.590159 6710 ts_tablet_manager.cc:902] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a: Initiating tablet copy from peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:16:07.604323 6710 tablet_copy_client.cc:308] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.198:43659
I20240502 14:16:07.605345 29329 tablet_copy_service.cc:140] P c1100e16b3b9495ca61d4f95e89b6a89: Received BeginTabletCopySession request for tablet 33a3639c26bf4b35a6e15d6ef173751c from peer ba187e38a2754d60a12de9ac4856430a ({username='slave'} at 127.19.94.194:42791)
I20240502 14:16:07.605535 29329 tablet_copy_service.cc:161] P c1100e16b3b9495ca61d4f95e89b6a89: Beginning new tablet copy session on tablet 33a3639c26bf4b35a6e15d6ef173751c from peer ba187e38a2754d60a12de9ac4856430a at {username='slave'} at 127.19.94.194:42791: session id = ba187e38a2754d60a12de9ac4856430a-33a3639c26bf4b35a6e15d6ef173751c
I20240502 14:16:07.608178 29329 tablet_copy_source_session.cc:205] T 33a3639c26bf4b35a6e15d6ef173751c P c1100e16b3b9495ca61d4f95e89b6a89: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:07.610860 6710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33a3639c26bf4b35a6e15d6ef173751c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:07.618227 6710 tablet_copy_client.cc:783] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:07.618533 6710 tablet_copy_client.cc:647] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:07.620767 6710 tablet_copy_client.cc:523] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:07.624491 6710 tablet_bootstrap.cc:492] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:16:07.637355 6710 tablet_bootstrap.cc:492] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:07.637872 6710 tablet_bootstrap.cc:492] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a: Bootstrap complete.
I20240502 14:16:07.638273 6710 ts_tablet_manager.cc:1367] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.007s
I20240502 14:16:07.639256 6710 raft_consensus.cc:348] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:16:07.639539 6710 raft_consensus.cc:729] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:16:07.639863 6710 consensus_queue.cc:260] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:16:07.640724 6710 ts_tablet_manager.cc:1398] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:16:07.641727 29329 tablet_copy_service.cc:342] P c1100e16b3b9495ca61d4f95e89b6a89: Request end of tablet copy session ba187e38a2754d60a12de9ac4856430a-33a3639c26bf4b35a6e15d6ef173751c received from {username='slave'} at 127.19.94.194:42791
I20240502 14:16:07.641875 29329 tablet_copy_service.cc:433] P c1100e16b3b9495ca61d4f95e89b6a89: ending tablet copy session ba187e38a2754d60a12de9ac4856430a-33a3639c26bf4b35a6e15d6ef173751c on tablet 33a3639c26bf4b35a6e15d6ef173751c with peer ba187e38a2754d60a12de9ac4856430a
I20240502 14:16:07.669250 29307 consensus_queue.cc:237] T 13136deb9046440f88bf3471ff42312c P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
W20240502 14:16:07.671947 29244 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P c1100e16b3b9495ca61d4f95e89b6a89 -> Peer 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): Couldn't send request to peer 89af7f423cf44717a08d6b142f0ed49d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 13136deb9046440f88bf3471ff42312c. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:07.671937 26558 raft_consensus.cc:1261] T 13136deb9046440f88bf3471ff42312c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer c1100e16b3b9495ca61d4f95e89b6a89: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:07.672022 29176 raft_consensus.cc:1261] T 13136deb9046440f88bf3471ff42312c P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer c1100e16b3b9495ca61d4f95e89b6a89: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:07.672524 6727 consensus_queue.cc:1035] T 13136deb9046440f88bf3471ff42312c P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:07.672848 5854 consensus_queue.cc:1035] T 13136deb9046440f88bf3471ff42312c P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:07.676517 5854 raft_consensus.cc:2932] T 13136deb9046440f88bf3471ff42312c P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:07.677767 29176 raft_consensus.cc:2932] T 13136deb9046440f88bf3471ff42312c P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:07.681762 26558 raft_consensus.cc:2932] T 13136deb9046440f88bf3471ff42312c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:07.685170 26283 catalog_manager.cc:5484] T 13136deb9046440f88bf3471ff42312c P c1100e16b3b9495ca61d4f95e89b6a89 reported cstate change: config changed from index -1 to 2, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New cstate: current_term: 1 leader_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:07.765678 29176 consensus_queue.cc:237] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
W20240502 14:16:07.770159 29112 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): Couldn't send request to peer 69ab78b2a1b74d2bbf2cdeafa0a4af23. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 253449163c65471d8736b92b25da0f92. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:07.771243 26558 raft_consensus.cc:1261] T 253449163c65471d8736b92b25da0f92 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:07.771754 6492 consensus_queue.cc:1035] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:07.771970 29307 raft_consensus.cc:1261] T 253449163c65471d8736b92b25da0f92 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:07.772886 6492 consensus_queue.cc:1035] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:07.775923 6105 raft_consensus.cc:2932] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:07.776690 29307 raft_consensus.cc:2932] T 253449163c65471d8736b92b25da0f92 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:07.783635 26558 raft_consensus.cc:2932] T 253449163c65471d8736b92b25da0f92 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:07.785943 26278 catalog_manager.cc:5484] T 253449163c65471d8736b92b25da0f92 P c1100e16b3b9495ca61d4f95e89b6a89 reported cstate change: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New cstate: current_term: 1 leader_uuid: "1cb67158133e4648a6eceba32d4dbba3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:07.824555 26427 raft_consensus.cc:1203] T 2e59791e95cc452ea30d1ed8bb6178ae P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:16:07.845891 6768 ts_tablet_manager.cc:902] T e45de4b8415446a08e9550f911367624 P 1cb67158133e4648a6eceba32d4dbba3: Initiating tablet copy from peer e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:16:07.847236 29575 consensus_queue.cc:237] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:16:07.847738 6768 tablet_copy_client.cc:308] T e45de4b8415446a08e9550f911367624 P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.196:32843
I20240502 14:16:07.848338 6690 ts_tablet_manager.cc:902] T aa4c1d9bf17a48b1b657bb7d03409efb P e4cfb8fc5a474177af6848a6efa13a27: Initiating tablet copy from peer ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:16:07.849061 26839 tablet_copy_service.cc:140] P e4cfb8fc5a474177af6848a6efa13a27: Received BeginTabletCopySession request for tablet e45de4b8415446a08e9550f911367624 from peer 1cb67158133e4648a6eceba32d4dbba3 ({username='slave'} at 127.19.94.197:46803)
I20240502 14:16:07.849200 26839 tablet_copy_service.cc:161] P e4cfb8fc5a474177af6848a6efa13a27: Beginning new tablet copy session on tablet e45de4b8415446a08e9550f911367624 from peer 1cb67158133e4648a6eceba32d4dbba3 at {username='slave'} at 127.19.94.197:46803: session id = 1cb67158133e4648a6eceba32d4dbba3-e45de4b8415446a08e9550f911367624
I20240502 14:16:07.853014 26839 tablet_copy_source_session.cc:205] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:07.855693 6768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e45de4b8415446a08e9550f911367624. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:07.861927 6768 tablet_copy_client.cc:783] T e45de4b8415446a08e9550f911367624 P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:07.862192 6768 tablet_copy_client.cc:647] T e45de4b8415446a08e9550f911367624 P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:07.863886 6768 tablet_copy_client.cc:523] T e45de4b8415446a08e9550f911367624 P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:07.864135 29307 raft_consensus.cc:1261] T 736ed8faceec4f0fa07c62952ea32bf6 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:07.864543 26558 raft_consensus.cc:1261] T 736ed8faceec4f0fa07c62952ea32bf6 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:07.864873 6165 consensus_queue.cc:1035] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:07.865263 6589 consensus_queue.cc:1035] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:16:07.866022 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 736ed8faceec4f0fa07c62952ea32bf6. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:07.867466 6768 tablet_bootstrap.cc:492] T e45de4b8415446a08e9550f911367624 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:16:07.880411 6768 tablet_bootstrap.cc:492] T e45de4b8415446a08e9550f911367624 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:07.880836 6768 tablet_bootstrap.cc:492] T e45de4b8415446a08e9550f911367624 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap complete.
I20240502 14:16:07.881172 6768 ts_tablet_manager.cc:1367] T e45de4b8415446a08e9550f911367624 P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.014s user 0.013s sys 0.000s
I20240502 14:16:07.882124 6768 raft_consensus.cc:348] T e45de4b8415446a08e9550f911367624 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:16:07.882432 6768 raft_consensus.cc:729] T e45de4b8415446a08e9550f911367624 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:16:07.882705 6768 consensus_queue.cc:260] T e45de4b8415446a08e9550f911367624 P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:16:07.883247 6768 ts_tablet_manager.cc:1398] T e45de4b8415446a08e9550f911367624 P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:16:07.883569 6690 tablet_copy_client.cc:308] T aa4c1d9bf17a48b1b657bb7d03409efb P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.194:36025
I20240502 14:16:07.884078 6165 raft_consensus.cc:2932] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:07.884486 26578 tablet_copy_service.cc:140] P ba187e38a2754d60a12de9ac4856430a: Received BeginTabletCopySession request for tablet aa4c1d9bf17a48b1b657bb7d03409efb from peer e4cfb8fc5a474177af6848a6efa13a27 ({username='slave'} at 127.19.94.196:51621)
I20240502 14:16:07.884670 26578 tablet_copy_service.cc:161] P ba187e38a2754d60a12de9ac4856430a: Beginning new tablet copy session on tablet aa4c1d9bf17a48b1b657bb7d03409efb from peer e4cfb8fc5a474177af6848a6efa13a27 at {username='slave'} at 127.19.94.196:51621: session id = e4cfb8fc5a474177af6848a6efa13a27-aa4c1d9bf17a48b1b657bb7d03409efb
I20240502 14:16:07.886917 29307 raft_consensus.cc:2932] T 736ed8faceec4f0fa07c62952ea32bf6 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:07.887682 26578 tablet_copy_source_session.cc:205] T aa4c1d9bf17a48b1b657bb7d03409efb P ba187e38a2754d60a12de9ac4856430a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:07.887698 26839 tablet_copy_service.cc:342] P e4cfb8fc5a474177af6848a6efa13a27: Request end of tablet copy session 1cb67158133e4648a6eceba32d4dbba3-e45de4b8415446a08e9550f911367624 received from {username='slave'} at 127.19.94.197:46803
I20240502 14:16:07.887823 26839 tablet_copy_service.cc:433] P e4cfb8fc5a474177af6848a6efa13a27: ending tablet copy session 1cb67158133e4648a6eceba32d4dbba3-e45de4b8415446a08e9550f911367624 on tablet e45de4b8415446a08e9550f911367624 with peer 1cb67158133e4648a6eceba32d4dbba3
I20240502 14:16:07.889200 6690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa4c1d9bf17a48b1b657bb7d03409efb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:07.891671 26558 raft_consensus.cc:2932] T 736ed8faceec4f0fa07c62952ea32bf6 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:07.902601 26280 catalog_manager.cc:5484] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: config changed from index -1 to 2, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:07.914086 6783 ts_tablet_manager.cc:902] T e1bdebd770244537a0119baa4e3f8091 P c1100e16b3b9495ca61d4f95e89b6a89: Initiating tablet copy from peer 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:16:07.922688 6690 tablet_copy_client.cc:783] T aa4c1d9bf17a48b1b657bb7d03409efb P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:07.923115 6690 tablet_copy_client.cc:647] T aa4c1d9bf17a48b1b657bb7d03409efb P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:07.931272 6690 tablet_copy_client.cc:523] T aa4c1d9bf17a48b1b657bb7d03409efb P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:07.944869 6783 tablet_copy_client.cc:308] T e1bdebd770244537a0119baa4e3f8091 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.193:37985
I20240502 14:16:07.945703 26447 tablet_copy_service.cc:140] P 89af7f423cf44717a08d6b142f0ed49d: Received BeginTabletCopySession request for tablet e1bdebd770244537a0119baa4e3f8091 from peer c1100e16b3b9495ca61d4f95e89b6a89 ({username='slave'} at 127.19.94.198:52561)
I20240502 14:16:07.945900 26447 tablet_copy_service.cc:161] P 89af7f423cf44717a08d6b142f0ed49d: Beginning new tablet copy session on tablet e1bdebd770244537a0119baa4e3f8091 from peer c1100e16b3b9495ca61d4f95e89b6a89 at {username='slave'} at 127.19.94.198:52561: session id = c1100e16b3b9495ca61d4f95e89b6a89-e1bdebd770244537a0119baa4e3f8091
I20240502 14:16:07.948531 6690 tablet_bootstrap.cc:492] T aa4c1d9bf17a48b1b657bb7d03409efb P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:16:07.948913 26447 tablet_copy_source_session.cc:205] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:07.950665 6783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1bdebd770244537a0119baa4e3f8091. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:07.957532 6783 tablet_copy_client.cc:783] T e1bdebd770244537a0119baa4e3f8091 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:07.957849 6783 tablet_copy_client.cc:647] T e1bdebd770244537a0119baa4e3f8091 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:07.963286 6783 tablet_copy_client.cc:523] T e1bdebd770244537a0119baa4e3f8091 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:07.972519 6783 tablet_bootstrap.cc:492] T e1bdebd770244537a0119baa4e3f8091 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:16:07.976025 6797 ts_tablet_manager.cc:902] T 451b303a32a3404d8d584203d6704b35 P 68a3c2b17ed94e06aba66ed4178132ab: Initiating tablet copy from peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:16:07.976270 6690 tablet_bootstrap.cc:492] T aa4c1d9bf17a48b1b657bb7d03409efb P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:07.976737 6690 tablet_bootstrap.cc:492] T aa4c1d9bf17a48b1b657bb7d03409efb P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap complete.
I20240502 14:16:07.977059 6690 ts_tablet_manager.cc:1367] T aa4c1d9bf17a48b1b657bb7d03409efb P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.029s user 0.010s sys 0.000s
I20240502 14:16:07.978102 6690 raft_consensus.cc:348] T aa4c1d9bf17a48b1b657bb7d03409efb P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:16:07.978468 6690 raft_consensus.cc:729] T aa4c1d9bf17a48b1b657bb7d03409efb P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:16:07.978765 6690 consensus_queue.cc:260] T aa4c1d9bf17a48b1b657bb7d03409efb P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:16:07.982393 6690 ts_tablet_manager.cc:1398] T aa4c1d9bf17a48b1b657bb7d03409efb P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240502 14:16:07.983280 26578 tablet_copy_service.cc:342] P ba187e38a2754d60a12de9ac4856430a: Request end of tablet copy session e4cfb8fc5a474177af6848a6efa13a27-aa4c1d9bf17a48b1b657bb7d03409efb received from {username='slave'} at 127.19.94.196:51621
I20240502 14:16:07.983440 26578 tablet_copy_service.cc:433] P ba187e38a2754d60a12de9ac4856430a: ending tablet copy session e4cfb8fc5a474177af6848a6efa13a27-aa4c1d9bf17a48b1b657bb7d03409efb on tablet aa4c1d9bf17a48b1b657bb7d03409efb with peer e4cfb8fc5a474177af6848a6efa13a27
I20240502 14:16:07.990120 6783 tablet_bootstrap.cc:492] T e1bdebd770244537a0119baa4e3f8091 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:07.990545 6783 tablet_bootstrap.cc:492] T e1bdebd770244537a0119baa4e3f8091 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap complete.
I20240502 14:16:07.990854 6783 ts_tablet_manager.cc:1367] T e1bdebd770244537a0119baa4e3f8091 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.004s
I20240502 14:16:07.991621 6783 raft_consensus.cc:348] T e1bdebd770244537a0119baa4e3f8091 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:16:07.991861 6783 raft_consensus.cc:729] T e1bdebd770244537a0119baa4e3f8091 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:16:07.992097 6783 consensus_queue.cc:260] T e1bdebd770244537a0119baa4e3f8091 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:16:07.992731 6783 ts_tablet_manager.cc:1398] T e1bdebd770244537a0119baa4e3f8091 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20240502 14:16:07.993907 26447 tablet_copy_service.cc:342] P 89af7f423cf44717a08d6b142f0ed49d: Request end of tablet copy session c1100e16b3b9495ca61d4f95e89b6a89-e1bdebd770244537a0119baa4e3f8091 received from {username='slave'} at 127.19.94.198:52561
I20240502 14:16:07.994072 26447 tablet_copy_service.cc:433] P 89af7f423cf44717a08d6b142f0ed49d: ending tablet copy session c1100e16b3b9495ca61d4f95e89b6a89-e1bdebd770244537a0119baa4e3f8091 on tablet e1bdebd770244537a0119baa4e3f8091 with peer c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:16:07.996712 26558 raft_consensus.cc:1203] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:16:08.017820 6797 tablet_copy_client.cc:308] T 451b303a32a3404d8d584203d6704b35 P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.195:44325
I20240502 14:16:08.018700 26709 tablet_copy_service.cc:140] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Received BeginTabletCopySession request for tablet 451b303a32a3404d8d584203d6704b35 from peer 68a3c2b17ed94e06aba66ed4178132ab ({username='slave'} at 127.19.94.199:51755)
I20240502 14:16:08.018879 26709 tablet_copy_service.cc:161] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Beginning new tablet copy session on tablet 451b303a32a3404d8d584203d6704b35 from peer 68a3c2b17ed94e06aba66ed4178132ab at {username='slave'} at 127.19.94.199:51755: session id = 68a3c2b17ed94e06aba66ed4178132ab-451b303a32a3404d8d584203d6704b35
I20240502 14:16:08.021209 26709 tablet_copy_source_session.cc:205] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:08.022562 6797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 451b303a32a3404d8d584203d6704b35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:08.030177 6797 tablet_copy_client.cc:783] T 451b303a32a3404d8d584203d6704b35 P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:08.030503 6797 tablet_copy_client.cc:647] T 451b303a32a3404d8d584203d6704b35 P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:08.035820 6797 tablet_copy_client.cc:523] T 451b303a32a3404d8d584203d6704b35 P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:08.039940 6797 tablet_bootstrap.cc:492] T 451b303a32a3404d8d584203d6704b35 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:16:08.056517 26687 consensus_queue.cc:237] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:16:08.058729 6797 tablet_bootstrap.cc:492] T 451b303a32a3404d8d584203d6704b35 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:08.059487 6797 tablet_bootstrap.cc:492] T 451b303a32a3404d8d584203d6704b35 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap complete.
I20240502 14:16:08.059864 6797 ts_tablet_manager.cc:1367] T 451b303a32a3404d8d584203d6704b35 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.020s user 0.018s sys 0.000s
I20240502 14:16:08.061249 6797 raft_consensus.cc:348] T 451b303a32a3404d8d584203d6704b35 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:16:08.061789 6797 raft_consensus.cc:729] T 451b303a32a3404d8d584203d6704b35 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:16:08.062141 6797 consensus_queue.cc:260] T 451b303a32a3404d8d584203d6704b35 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:16:08.063035 6797 ts_tablet_manager.cc:1398] T 451b303a32a3404d8d584203d6704b35 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:16:08.065676 26709 tablet_copy_service.cc:342] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Request end of tablet copy session 68a3c2b17ed94e06aba66ed4178132ab-451b303a32a3404d8d584203d6704b35 received from {username='slave'} at 127.19.94.199:51755
I20240502 14:16:08.065814 26709 tablet_copy_service.cc:433] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: ending tablet copy session 68a3c2b17ed94e06aba66ed4178132ab-451b303a32a3404d8d584203d6704b35 on tablet 451b303a32a3404d8d584203d6704b35 with peer 68a3c2b17ed94e06aba66ed4178132ab
I20240502 14:16:08.069869 26818 raft_consensus.cc:1261] T 0290ebbcf609427cae8fce6637019336 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:08.071419 6312 consensus_queue.cc:1035] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:16:08.076344 26625 consensus_peers.cc:485] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097): Couldn't send request to peer 4092902b3e674857a0147f1b4f2cdb86. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0290ebbcf609427cae8fce6637019336. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:08.077901 26427 raft_consensus.cc:1261] T 0290ebbcf609427cae8fce6637019336 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:08.078655 6312 consensus_queue.cc:1035] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:08.082749 6701 raft_consensus.cc:2932] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:16:08.083465 26427 raft_consensus.cc:2932] T 0290ebbcf609427cae8fce6637019336 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:16:08.093537 26280 catalog_manager.cc:5484] T 0290ebbcf609427cae8fce6637019336 P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: config changed from index -1 to 4, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:16:08.095444 6814 ts_tablet_manager.cc:902] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d: Initiating tablet copy from peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:16:08.095754 26818 raft_consensus.cc:2932] T 0290ebbcf609427cae8fce6637019336 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:16:08.111557 26818 raft_consensus.cc:1203] T 83dd34749ff8496483365b30a86db71f P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:16:08.140425 6814 tablet_copy_client.cc:308] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.198:43659
I20240502 14:16:08.141594 29329 tablet_copy_service.cc:140] P c1100e16b3b9495ca61d4f95e89b6a89: Received BeginTabletCopySession request for tablet 13136deb9046440f88bf3471ff42312c from peer 89af7f423cf44717a08d6b142f0ed49d ({username='slave'} at 127.19.94.193:58815)
I20240502 14:16:08.141834 29329 tablet_copy_service.cc:161] P c1100e16b3b9495ca61d4f95e89b6a89: Beginning new tablet copy session on tablet 13136deb9046440f88bf3471ff42312c from peer 89af7f423cf44717a08d6b142f0ed49d at {username='slave'} at 127.19.94.193:58815: session id = 89af7f423cf44717a08d6b142f0ed49d-13136deb9046440f88bf3471ff42312c
I20240502 14:16:08.146138 29329 tablet_copy_source_session.cc:205] T 13136deb9046440f88bf3471ff42312c P c1100e16b3b9495ca61d4f95e89b6a89: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:08.148903 6814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13136deb9046440f88bf3471ff42312c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:08.160006 6814 tablet_copy_client.cc:783] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:08.161375 6814 tablet_copy_client.cc:647] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:08.167009 6814 tablet_copy_client.cc:523] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:08.167271 6829 ts_tablet_manager.cc:902] T 253449163c65471d8736b92b25da0f92 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Initiating tablet copy from peer 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:16:08.171949 6814 tablet_bootstrap.cc:492] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:16:08.197109 6814 tablet_bootstrap.cc:492] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:08.197958 6814 tablet_bootstrap.cc:492] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap complete.
I20240502 14:16:08.198345 6814 ts_tablet_manager.cc:1367] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.027s user 0.012s sys 0.006s
I20240502 14:16:08.199985 6829 tablet_copy_client.cc:308] T 253449163c65471d8736b92b25da0f92 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.197:38715
I20240502 14:16:08.200238 6814 raft_consensus.cc:348] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:08.200701 6814 raft_consensus.cc:729] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:16:08.200719 29196 tablet_copy_service.cc:140] P 1cb67158133e4648a6eceba32d4dbba3: Received BeginTabletCopySession request for tablet 253449163c65471d8736b92b25da0f92 from peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 ({username='slave'} at 127.19.94.195:56139)
I20240502 14:16:08.200867 29196 tablet_copy_service.cc:161] P 1cb67158133e4648a6eceba32d4dbba3: Beginning new tablet copy session on tablet 253449163c65471d8736b92b25da0f92 from peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 at {username='slave'} at 127.19.94.195:56139: session id = 69ab78b2a1b74d2bbf2cdeafa0a4af23-253449163c65471d8736b92b25da0f92
I20240502 14:16:08.200991 6814 consensus_queue.cc:260] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:08.201743 6814 ts_tablet_manager.cc:1398] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:16:08.203466 29196 tablet_copy_source_session.cc:205] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:08.203550 29329 tablet_copy_service.cc:342] P c1100e16b3b9495ca61d4f95e89b6a89: Request end of tablet copy session 89af7f423cf44717a08d6b142f0ed49d-13136deb9046440f88bf3471ff42312c received from {username='slave'} at 127.19.94.193:58815
I20240502 14:16:08.203667 29329 tablet_copy_service.cc:433] P c1100e16b3b9495ca61d4f95e89b6a89: ending tablet copy session 89af7f423cf44717a08d6b142f0ed49d-13136deb9046440f88bf3471ff42312c on tablet 13136deb9046440f88bf3471ff42312c with peer 89af7f423cf44717a08d6b142f0ed49d
I20240502 14:16:08.206562 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 253449163c65471d8736b92b25da0f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:08.216007 6829 tablet_copy_client.cc:783] T 253449163c65471d8736b92b25da0f92 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:08.216425 6829 tablet_copy_client.cc:647] T 253449163c65471d8736b92b25da0f92 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:08.224520 6829 tablet_copy_client.cc:523] T 253449163c65471d8736b92b25da0f92 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:08.227962 26818 raft_consensus.cc:1203] T aa4c1d9bf17a48b1b657bb7d03409efb P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:16:08.229207 6829 tablet_bootstrap.cc:492] T 253449163c65471d8736b92b25da0f92 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:16:08.244482 6829 tablet_bootstrap.cc:492] T 253449163c65471d8736b92b25da0f92 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:08.244957 6829 tablet_bootstrap.cc:492] T 253449163c65471d8736b92b25da0f92 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap complete.
I20240502 14:16:08.245235 6829 ts_tablet_manager.cc:1367] T 253449163c65471d8736b92b25da0f92 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.007s
I20240502 14:16:08.246023 6829 raft_consensus.cc:348] T 253449163c65471d8736b92b25da0f92 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:16:08.246259 6829 raft_consensus.cc:729] T 253449163c65471d8736b92b25da0f92 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:16:08.246519 6829 consensus_queue.cc:260] T 253449163c65471d8736b92b25da0f92 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:16:08.247073 6829 ts_tablet_manager.cc:1398] T 253449163c65471d8736b92b25da0f92 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:16:08.255570 29196 tablet_copy_service.cc:342] P 1cb67158133e4648a6eceba32d4dbba3: Request end of tablet copy session 69ab78b2a1b74d2bbf2cdeafa0a4af23-253449163c65471d8736b92b25da0f92 received from {username='slave'} at 127.19.94.195:56139
I20240502 14:16:08.255724 29196 tablet_copy_service.cc:433] P 1cb67158133e4648a6eceba32d4dbba3: ending tablet copy session 69ab78b2a1b74d2bbf2cdeafa0a4af23-253449163c65471d8736b92b25da0f92 on tablet 253449163c65471d8736b92b25da0f92 with peer 69ab78b2a1b74d2bbf2cdeafa0a4af23
I20240502 14:16:08.280498 29575 consensus_queue.cc:237] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:08.282956 29307 raft_consensus.cc:1261] T c115987981694996abe797e03b27f799 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:08.283501 29174 raft_consensus.cc:1261] T c115987981694996abe797e03b27f799 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:16:08.283890 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): Couldn't send request to peer 89af7f423cf44717a08d6b142f0ed49d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c115987981694996abe797e03b27f799. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:08.284572 6165 consensus_queue.cc:1035] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:08.285271 6781 consensus_queue.cc:1035] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:08.288255 6165 raft_consensus.cc:2932] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:08.288874 29174 raft_consensus.cc:2932] T c115987981694996abe797e03b27f799 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:08.294468 29307 raft_consensus.cc:2932] T c115987981694996abe797e03b27f799 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:08.295141 26280 catalog_manager.cc:5484] T c115987981694996abe797e03b27f799 P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: config changed from index -1 to 2, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:08.299325 29174 raft_consensus.cc:1203] T e45de4b8415446a08e9550f911367624 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:16:08.302734 6797 ts_tablet_manager.cc:902] T 736ed8faceec4f0fa07c62952ea32bf6 P 68a3c2b17ed94e06aba66ed4178132ab: Initiating tablet copy from peer 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:16:08.323904 6797 tablet_copy_client.cc:308] T 736ed8faceec4f0fa07c62952ea32bf6 P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.200:46097
I20240502 14:16:08.324749 29594 tablet_copy_service.cc:140] P 4092902b3e674857a0147f1b4f2cdb86: Received BeginTabletCopySession request for tablet 736ed8faceec4f0fa07c62952ea32bf6 from peer 68a3c2b17ed94e06aba66ed4178132ab ({username='slave'} at 127.19.94.199:59271)
I20240502 14:16:08.324930 29594 tablet_copy_service.cc:161] P 4092902b3e674857a0147f1b4f2cdb86: Beginning new tablet copy session on tablet 736ed8faceec4f0fa07c62952ea32bf6 from peer 68a3c2b17ed94e06aba66ed4178132ab at {username='slave'} at 127.19.94.199:59271: session id = 68a3c2b17ed94e06aba66ed4178132ab-736ed8faceec4f0fa07c62952ea32bf6
I20240502 14:16:08.327433 29594 tablet_copy_source_session.cc:205] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:08.329003 6797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 736ed8faceec4f0fa07c62952ea32bf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:08.335500 6797 tablet_copy_client.cc:783] T 736ed8faceec4f0fa07c62952ea32bf6 P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:08.335764 6797 tablet_copy_client.cc:647] T 736ed8faceec4f0fa07c62952ea32bf6 P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:08.338260 6797 tablet_copy_client.cc:523] T 736ed8faceec4f0fa07c62952ea32bf6 P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:08.341893 6797 tablet_bootstrap.cc:492] T 736ed8faceec4f0fa07c62952ea32bf6 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:16:08.348871 6797 tablet_bootstrap.cc:492] T 736ed8faceec4f0fa07c62952ea32bf6 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:08.349313 6797 tablet_bootstrap.cc:492] T 736ed8faceec4f0fa07c62952ea32bf6 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap complete.
I20240502 14:16:08.349639 6797 ts_tablet_manager.cc:1367] T 736ed8faceec4f0fa07c62952ea32bf6 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240502 14:16:08.350320 6797 raft_consensus.cc:348] T 736ed8faceec4f0fa07c62952ea32bf6 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:16:08.350579 6797 raft_consensus.cc:729] T 736ed8faceec4f0fa07c62952ea32bf6 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:16:08.350817 6797 consensus_queue.cc:260] T 736ed8faceec4f0fa07c62952ea32bf6 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:16:08.353024 6797 ts_tablet_manager.cc:1398] T 736ed8faceec4f0fa07c62952ea32bf6 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 14:16:08.353729 29594 tablet_copy_service.cc:342] P 4092902b3e674857a0147f1b4f2cdb86: Request end of tablet copy session 68a3c2b17ed94e06aba66ed4178132ab-736ed8faceec4f0fa07c62952ea32bf6 received from {username='slave'} at 127.19.94.199:59271
I20240502 14:16:08.353866 29594 tablet_copy_service.cc:433] P 4092902b3e674857a0147f1b4f2cdb86: ending tablet copy session 68a3c2b17ed94e06aba66ed4178132ab-736ed8faceec4f0fa07c62952ea32bf6 on tablet 736ed8faceec4f0fa07c62952ea32bf6 with peer 68a3c2b17ed94e06aba66ed4178132ab
I20240502 14:16:08.422744 29309 raft_consensus.cc:1203] T e1bdebd770244537a0119baa4e3f8091 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:16:08.435990 29440 raft_consensus.cc:1203] T 451b303a32a3404d8d584203d6704b35 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:16:08.460968 26818 consensus_queue.cc:237] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:16:08.464377 26687 raft_consensus.cc:1261] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:08.464694 26556 raft_consensus.cc:1261] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:08.465183 6206 consensus_queue.cc:1035] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:16:08.465483 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c9fb1f7eb71c42d2b8f0f6aacd2b0aeb. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:08.465965 6629 consensus_queue.cc:1035] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:08.471127 6206 raft_consensus.cc:2932] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:08.472258 26686 raft_consensus.cc:2932] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:08.480547 26556 raft_consensus.cc:2932] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:08.481693 26278 catalog_manager.cc:5484] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: config changed from index -1 to 4, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:08.500808 6869 ts_tablet_manager.cc:902] T 0290ebbcf609427cae8fce6637019336 P 4092902b3e674857a0147f1b4f2cdb86: Initiating tablet copy from peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:16:08.502339 6869 tablet_copy_client.cc:308] T 0290ebbcf609427cae8fce6637019336 P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.195:44325
I20240502 14:16:08.503765 26709 tablet_copy_service.cc:140] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Received BeginTabletCopySession request for tablet 0290ebbcf609427cae8fce6637019336 from peer 4092902b3e674857a0147f1b4f2cdb86 ({username='slave'} at 127.19.94.200:47557)
I20240502 14:16:08.504032 26709 tablet_copy_service.cc:161] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Beginning new tablet copy session on tablet 0290ebbcf609427cae8fce6637019336 from peer 4092902b3e674857a0147f1b4f2cdb86 at {username='slave'} at 127.19.94.200:47557: session id = 4092902b3e674857a0147f1b4f2cdb86-0290ebbcf609427cae8fce6637019336
I20240502 14:16:08.506963 26709 tablet_copy_source_session.cc:205] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:08.516789 6869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0290ebbcf609427cae8fce6637019336. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:08.530375 6869 tablet_copy_client.cc:783] T 0290ebbcf609427cae8fce6637019336 P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:08.530812 6869 tablet_copy_client.cc:647] T 0290ebbcf609427cae8fce6637019336 P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:08.532692 6869 tablet_copy_client.cc:523] T 0290ebbcf609427cae8fce6637019336 P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:08.541060 6869 tablet_bootstrap.cc:492] T 0290ebbcf609427cae8fce6637019336 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:16:08.581897 26556 consensus_queue.cc:237] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:16:08.588055 26818 raft_consensus.cc:1261] T 4bf6e7410afa4c048f5842ec2a426d77 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:08.588696 26686 raft_consensus.cc:1261] T 4bf6e7410afa4c048f5842ec2a426d77 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:08.588819 6869 tablet_bootstrap.cc:492] T 0290ebbcf609427cae8fce6637019336 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:08.588714 6674 consensus_queue.cc:1035] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:08.589267 6869 tablet_bootstrap.cc:492] T 0290ebbcf609427cae8fce6637019336 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap complete.
I20240502 14:16:08.589552 6869 ts_tablet_manager.cc:1367] T 0290ebbcf609427cae8fce6637019336 P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.049s user 0.038s sys 0.005s
W20240502 14:16:08.590188 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4bf6e7410afa4c048f5842ec2a426d77. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:08.590140 6869 raft_consensus.cc:348] T 0290ebbcf609427cae8fce6637019336 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:16:08.590366 6869 raft_consensus.cc:729] T 0290ebbcf609427cae8fce6637019336 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:16:08.590509 6674 consensus_queue.cc:1035] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:08.590612 6869 consensus_queue.cc:260] T 0290ebbcf609427cae8fce6637019336 P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:16:08.591202 6869 ts_tablet_manager.cc:1398] T 0290ebbcf609427cae8fce6637019336 P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:16:08.592614 26709 tablet_copy_service.cc:342] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Request end of tablet copy session 4092902b3e674857a0147f1b4f2cdb86-0290ebbcf609427cae8fce6637019336 received from {username='slave'} at 127.19.94.200:47557
I20240502 14:16:08.592765 26709 tablet_copy_service.cc:433] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: ending tablet copy session 4092902b3e674857a0147f1b4f2cdb86-0290ebbcf609427cae8fce6637019336 on tablet 0290ebbcf609427cae8fce6637019336 with peer 4092902b3e674857a0147f1b4f2cdb86
I20240502 14:16:08.595413 6674 raft_consensus.cc:2932] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:08.600517 26686 raft_consensus.cc:2932] T 4bf6e7410afa4c048f5842ec2a426d77 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:08.612179 26283 catalog_manager.cc:5484] T 4bf6e7410afa4c048f5842ec2a426d77 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: config changed from index -1 to 4, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:08.614277 26818 raft_consensus.cc:2932] T 4bf6e7410afa4c048f5842ec2a426d77 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:08.676337 26686 raft_consensus.cc:1203] T 253449163c65471d8736b92b25da0f92 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:16:08.682353 26427 consensus_queue.cc:237] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:16:08.684181 26425 raft_consensus.cc:1203] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:16:08.698896 26818 raft_consensus.cc:1261] T d16a118ea4c04a58890c7dd3ff9ec7b4 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:16:08.699155 26363 consensus_peers.cc:485] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097): Couldn't send request to peer 4092902b3e674857a0147f1b4f2cdb86. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d16a118ea4c04a58890c7dd3ff9ec7b4. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:08.699491 6096 consensus_queue.cc:1035] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:08.701771 26686 raft_consensus.cc:1261] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:08.702365 6096 consensus_queue.cc:1035] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:08.706029 6912 raft_consensus.cc:2932] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:16:08.706902 26686 raft_consensus.cc:2932] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:16:08.713752 26818 raft_consensus.cc:2932] T d16a118ea4c04a58890c7dd3ff9ec7b4 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:16:08.715070 26280 catalog_manager.cc:5484] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: config changed from index -1 to 2, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:16:08.752823 6921 ts_tablet_manager.cc:902] T c115987981694996abe797e03b27f799 P 89af7f423cf44717a08d6b142f0ed49d: Initiating tablet copy from peer 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:16:08.774010 6921 tablet_copy_client.cc:308] T c115987981694996abe797e03b27f799 P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.200:46097
I20240502 14:16:08.776897 29594 tablet_copy_service.cc:140] P 4092902b3e674857a0147f1b4f2cdb86: Received BeginTabletCopySession request for tablet c115987981694996abe797e03b27f799 from peer 89af7f423cf44717a08d6b142f0ed49d ({username='slave'} at 127.19.94.193:43933)
I20240502 14:16:08.777114 29594 tablet_copy_service.cc:161] P 4092902b3e674857a0147f1b4f2cdb86: Beginning new tablet copy session on tablet c115987981694996abe797e03b27f799 from peer 89af7f423cf44717a08d6b142f0ed49d at {username='slave'} at 127.19.94.193:43933: session id = 89af7f423cf44717a08d6b142f0ed49d-c115987981694996abe797e03b27f799
I20240502 14:16:08.780192 29594 tablet_copy_source_session.cc:205] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:08.782399 6921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c115987981694996abe797e03b27f799. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:08.797379 6921 tablet_copy_client.cc:783] T c115987981694996abe797e03b27f799 P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:08.797637 6921 tablet_copy_client.cc:647] T c115987981694996abe797e03b27f799 P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:08.802675 6921 tablet_copy_client.cc:523] T c115987981694996abe797e03b27f799 P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:08.807170 6921 tablet_bootstrap.cc:492] T c115987981694996abe797e03b27f799 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:16:08.821194 6921 tablet_bootstrap.cc:492] T c115987981694996abe797e03b27f799 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:08.821631 6921 tablet_bootstrap.cc:492] T c115987981694996abe797e03b27f799 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap complete.
I20240502 14:16:08.821892 6921 ts_tablet_manager.cc:1367] T c115987981694996abe797e03b27f799 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.015s user 0.013s sys 0.001s
I20240502 14:16:08.822666 6921 raft_consensus.cc:348] T c115987981694996abe797e03b27f799 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:08.822926 6921 raft_consensus.cc:729] T c115987981694996abe797e03b27f799 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:16:08.823185 6921 consensus_queue.cc:260] T c115987981694996abe797e03b27f799 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:08.823778 6921 ts_tablet_manager.cc:1398] T c115987981694996abe797e03b27f799 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:16:08.826354 29594 tablet_copy_service.cc:342] P 4092902b3e674857a0147f1b4f2cdb86: Request end of tablet copy session 89af7f423cf44717a08d6b142f0ed49d-c115987981694996abe797e03b27f799 received from {username='slave'} at 127.19.94.193:43933
I20240502 14:16:08.826510 29594 tablet_copy_service.cc:433] P 4092902b3e674857a0147f1b4f2cdb86: ending tablet copy session 89af7f423cf44717a08d6b142f0ed49d-c115987981694996abe797e03b27f799 on tablet c115987981694996abe797e03b27f799 with peer 89af7f423cf44717a08d6b142f0ed49d
I20240502 14:16:08.865523 29442 raft_consensus.cc:1203] T 736ed8faceec4f0fa07c62952ea32bf6 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:16:08.911093 6949 ts_tablet_manager.cc:902] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P c1100e16b3b9495ca61d4f95e89b6a89: Initiating tablet copy from peer e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:16:08.925612 6949 tablet_copy_client.cc:308] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.196:32843
I20240502 14:16:08.932981 26839 tablet_copy_service.cc:140] P e4cfb8fc5a474177af6848a6efa13a27: Received BeginTabletCopySession request for tablet c9fb1f7eb71c42d2b8f0f6aacd2b0aeb from peer c1100e16b3b9495ca61d4f95e89b6a89 ({username='slave'} at 127.19.94.198:38611)
I20240502 14:16:08.933183 26839 tablet_copy_service.cc:161] P e4cfb8fc5a474177af6848a6efa13a27: Beginning new tablet copy session on tablet c9fb1f7eb71c42d2b8f0f6aacd2b0aeb from peer c1100e16b3b9495ca61d4f95e89b6a89 at {username='slave'} at 127.19.94.198:38611: session id = c1100e16b3b9495ca61d4f95e89b6a89-c9fb1f7eb71c42d2b8f0f6aacd2b0aeb
I20240502 14:16:08.935999 26839 tablet_copy_source_session.cc:205] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:08.939846 6949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9fb1f7eb71c42d2b8f0f6aacd2b0aeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:08.947746 6949 tablet_copy_client.cc:783] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:08.948698 6949 tablet_copy_client.cc:647] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:08.951675 6949 tablet_copy_client.cc:523] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:08.955157 6949 tablet_bootstrap.cc:492] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:16:08.985543 29573 raft_consensus.cc:1203] T 0290ebbcf609427cae8fce6637019336 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:16:09.022032 6949 tablet_bootstrap.cc:492] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:09.022725 6949 tablet_bootstrap.cc:492] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap complete.
I20240502 14:16:09.023116 6949 ts_tablet_manager.cc:1367] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.068s user 0.050s sys 0.005s
I20240502 14:16:09.024153 6949 raft_consensus.cc:348] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:16:09.024585 6949 raft_consensus.cc:729] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:16:09.024936 6949 consensus_queue.cc:260] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:16:09.027134 6949 ts_tablet_manager.cc:1398] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:16:09.028038 26839 tablet_copy_service.cc:342] P e4cfb8fc5a474177af6848a6efa13a27: Request end of tablet copy session c1100e16b3b9495ca61d4f95e89b6a89-c9fb1f7eb71c42d2b8f0f6aacd2b0aeb received from {username='slave'} at 127.19.94.198:38611
I20240502 14:16:09.028174 26839 tablet_copy_service.cc:433] P e4cfb8fc5a474177af6848a6efa13a27: ending tablet copy session c1100e16b3b9495ca61d4f95e89b6a89-c9fb1f7eb71c42d2b8f0f6aacd2b0aeb on tablet c9fb1f7eb71c42d2b8f0f6aacd2b0aeb with peer c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:16:09.175218 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:09.182997 6994 ts_tablet_manager.cc:902] T 4bf6e7410afa4c048f5842ec2a426d77 P 68a3c2b17ed94e06aba66ed4178132ab: Initiating tablet copy from peer ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:16:09.200068 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:16:09.203434 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:09.208488 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:09.244787 26427 raft_consensus.cc:1203] T c115987981694996abe797e03b27f799 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:16:09.248090 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:09.257133 6994 tablet_copy_client.cc:308] T 4bf6e7410afa4c048f5842ec2a426d77 P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.194:36025
I20240502 14:16:09.258219 26578 tablet_copy_service.cc:140] P ba187e38a2754d60a12de9ac4856430a: Received BeginTabletCopySession request for tablet 4bf6e7410afa4c048f5842ec2a426d77 from peer 68a3c2b17ed94e06aba66ed4178132ab ({username='slave'} at 127.19.94.199:55497)
I20240502 14:16:09.258394 26578 tablet_copy_service.cc:161] P ba187e38a2754d60a12de9ac4856430a: Beginning new tablet copy session on tablet 4bf6e7410afa4c048f5842ec2a426d77 from peer 68a3c2b17ed94e06aba66ed4178132ab at {username='slave'} at 127.19.94.199:55497: session id = 68a3c2b17ed94e06aba66ed4178132ab-4bf6e7410afa4c048f5842ec2a426d77
I20240502 14:16:09.262001 26578 tablet_copy_source_session.cc:205] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:09.266939 6994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bf6e7410afa4c048f5842ec2a426d77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:09.287312 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:09.291316 6994 tablet_copy_client.cc:783] T 4bf6e7410afa4c048f5842ec2a426d77 P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:09.291656 6994 tablet_copy_client.cc:647] T 4bf6e7410afa4c048f5842ec2a426d77 P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:09.296432 7001 ts_tablet_manager.cc:902] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 4092902b3e674857a0147f1b4f2cdb86: Initiating tablet copy from peer 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:16:09.297184 6994 tablet_copy_client.cc:523] T 4bf6e7410afa4c048f5842ec2a426d77 P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:09.300886 6994 tablet_bootstrap.cc:492] T 4bf6e7410afa4c048f5842ec2a426d77 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:16:09.312249 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:09.408054 6994 tablet_bootstrap.cc:492] T 4bf6e7410afa4c048f5842ec2a426d77 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:09.408571 6994 tablet_bootstrap.cc:492] T 4bf6e7410afa4c048f5842ec2a426d77 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap complete.
I20240502 14:16:09.410612 6994 ts_tablet_manager.cc:1367] T 4bf6e7410afa4c048f5842ec2a426d77 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.110s user 0.039s sys 0.008s
I20240502 14:16:09.411413 6994 raft_consensus.cc:348] T 4bf6e7410afa4c048f5842ec2a426d77 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:16:09.411716 6994 raft_consensus.cc:729] T 4bf6e7410afa4c048f5842ec2a426d77 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:16:09.411975 6994 consensus_queue.cc:260] T 4bf6e7410afa4c048f5842ec2a426d77 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:16:09.412549 6994 ts_tablet_manager.cc:1398] T 4bf6e7410afa4c048f5842ec2a426d77 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:16:09.416131 26578 tablet_copy_service.cc:342] P ba187e38a2754d60a12de9ac4856430a: Request end of tablet copy session 68a3c2b17ed94e06aba66ed4178132ab-4bf6e7410afa4c048f5842ec2a426d77 received from {username='slave'} at 127.19.94.199:55497
I20240502 14:16:09.416275 26578 tablet_copy_service.cc:433] P ba187e38a2754d60a12de9ac4856430a: ending tablet copy session 68a3c2b17ed94e06aba66ed4178132ab-4bf6e7410afa4c048f5842ec2a426d77 on tablet 4bf6e7410afa4c048f5842ec2a426d77 with peer 68a3c2b17ed94e06aba66ed4178132ab
I20240502 14:16:09.434401 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:09.456488 29307 raft_consensus.cc:1203] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:16:09.540575 7001 tablet_copy_client.cc:308] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.193:37985
I20240502 14:16:09.541399 26447 tablet_copy_service.cc:140] P 89af7f423cf44717a08d6b142f0ed49d: Received BeginTabletCopySession request for tablet d16a118ea4c04a58890c7dd3ff9ec7b4 from peer 4092902b3e674857a0147f1b4f2cdb86 ({username='slave'} at 127.19.94.200:60351)
I20240502 14:16:09.541584 26447 tablet_copy_service.cc:161] P 89af7f423cf44717a08d6b142f0ed49d: Beginning new tablet copy session on tablet d16a118ea4c04a58890c7dd3ff9ec7b4 from peer 4092902b3e674857a0147f1b4f2cdb86 at {username='slave'} at 127.19.94.200:60351: session id = 4092902b3e674857a0147f1b4f2cdb86-d16a118ea4c04a58890c7dd3ff9ec7b4
I20240502 14:16:09.545094 26447 tablet_copy_source_session.cc:205] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 89af7f423cf44717a08d6b142f0ed49d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:09.547722 7001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d16a118ea4c04a58890c7dd3ff9ec7b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:09.554998 7001 tablet_copy_client.cc:783] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:09.555289 7001 tablet_copy_client.cc:647] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:09.557215 7001 tablet_copy_client.cc:523] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:09.561129 7001 tablet_bootstrap.cc:492] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:16:09.564195 29440 raft_consensus.cc:1203] T 4bf6e7410afa4c048f5842ec2a426d77 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:16:09.569594 7001 tablet_bootstrap.cc:492] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:09.570144 7001 tablet_bootstrap.cc:492] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap complete.
I20240502 14:16:09.570583 7001 ts_tablet_manager.cc:1367] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.002s
I20240502 14:16:09.573259 7001 raft_consensus.cc:348] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:16:09.573549 7001 raft_consensus.cc:729] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:16:09.573798 7001 consensus_queue.cc:260] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:16:09.574330 7001 ts_tablet_manager.cc:1398] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:16:09.577090 26447 tablet_copy_service.cc:342] P 89af7f423cf44717a08d6b142f0ed49d: Request end of tablet copy session 4092902b3e674857a0147f1b4f2cdb86-d16a118ea4c04a58890c7dd3ff9ec7b4 received from {username='slave'} at 127.19.94.200:60351
I20240502 14:16:09.577227 26447 tablet_copy_service.cc:433] P 89af7f423cf44717a08d6b142f0ed49d: ending tablet copy session 4092902b3e674857a0147f1b4f2cdb86-d16a118ea4c04a58890c7dd3ff9ec7b4 on tablet d16a118ea4c04a58890c7dd3ff9ec7b4 with peer 4092902b3e674857a0147f1b4f2cdb86
I20240502 14:16:09.857614 29573 raft_consensus.cc:1203] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:16:09.876045 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:09.929033 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:09.929468 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:16:09.940693 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:09.955655 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:09.957974 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:10.015905 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:10.022218 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:10.239135 29573 consensus_queue.cc:237] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:10.241035 26425 raft_consensus.cc:1261] T c115987981694996abe797e03b27f799 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:10.241190 29309 raft_consensus.cc:1261] T c115987981694996abe797e03b27f799 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:10.241765 6952 consensus_queue.cc:1035] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:10.242206 6165 consensus_queue.cc:1035] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:10.245940 6165 raft_consensus.cc:2932] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:10.246692 29309 raft_consensus.cc:2932] T c115987981694996abe797e03b27f799 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:10.247608 26425 raft_consensus.cc:2932] T c115987981694996abe797e03b27f799 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:10.256018 26282 catalog_manager.cc:5484] T c115987981694996abe797e03b27f799 P c1100e16b3b9495ca61d4f95e89b6a89 reported cstate change: config changed from index 2 to 3, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) evicted. New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:10.275058 29155 tablet_service.cc:1514] Processing DeleteTablet for tablet c115987981694996abe797e03b27f799 with delete_type TABLET_DATA_TOMBSTONED (TS 1cb67158133e4648a6eceba32d4dbba3 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:55202
I20240502 14:16:10.276116 7103 tablet_replica.cc:330] T c115987981694996abe797e03b27f799 P 1cb67158133e4648a6eceba32d4dbba3: stopping tablet replica
I20240502 14:16:10.276355 7103 raft_consensus.cc:2229] T c115987981694996abe797e03b27f799 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:10.276633 7103 raft_consensus.cc:2258] T c115987981694996abe797e03b27f799 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:10.278584 7103 ts_tablet_manager.cc:1875] T c115987981694996abe797e03b27f799 P 1cb67158133e4648a6eceba32d4dbba3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:10.288184 7103 ts_tablet_manager.cc:1888] T c115987981694996abe797e03b27f799 P 1cb67158133e4648a6eceba32d4dbba3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:16:10.290406 7103 log.cc:1196] T c115987981694996abe797e03b27f799 P 1cb67158133e4648a6eceba32d4dbba3: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal/wals/c115987981694996abe797e03b27f799
I20240502 14:16:10.291632 26271 catalog_manager.cc:4830] TS 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): tablet c115987981694996abe797e03b27f799 (table rebalancer_extra_table_34 [id=b4142465f8a74e2baf1a95d85f3fd781]) successfully deleted
I20240502 14:16:10.674589 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:10.703666 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:10.704770 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:10.723652 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:10.762264 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:10.766284 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:16:10.818897 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:10.819756 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:10.993131 29575 consensus_queue.cc:237] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:16:10.994982 29442 raft_consensus.cc:1261] T 736ed8faceec4f0fa07c62952ea32bf6 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:10.995155 26556 raft_consensus.cc:1261] T 736ed8faceec4f0fa07c62952ea32bf6 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:10.995488 7096 consensus_queue.cc:1035] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:10.995797 6952 consensus_queue.cc:1035] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:10.998672 6165 raft_consensus.cc:2932] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:10.999986 29442 raft_consensus.cc:2932] T 736ed8faceec4f0fa07c62952ea32bf6 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:11.002929 26556 raft_consensus.cc:2932] T 736ed8faceec4f0fa07c62952ea32bf6 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:11.004683 26278 catalog_manager.cc:5484] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:11.026859 29287 tablet_service.cc:1514] Processing DeleteTablet for tablet 736ed8faceec4f0fa07c62952ea32bf6 with delete_type TABLET_DATA_TOMBSTONED (TS c1100e16b3b9495ca61d4f95e89b6a89 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:59160
I20240502 14:16:11.040578 7196 tablet_replica.cc:330] T 736ed8faceec4f0fa07c62952ea32bf6 P c1100e16b3b9495ca61d4f95e89b6a89: stopping tablet replica
I20240502 14:16:11.041019 7196 raft_consensus.cc:2229] T 736ed8faceec4f0fa07c62952ea32bf6 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:11.043598 7196 raft_consensus.cc:2258] T 736ed8faceec4f0fa07c62952ea32bf6 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:11.044425 7196 ts_tablet_manager.cc:1875] T 736ed8faceec4f0fa07c62952ea32bf6 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:11.053723 7196 ts_tablet_manager.cc:1888] T 736ed8faceec4f0fa07c62952ea32bf6 P c1100e16b3b9495ca61d4f95e89b6a89: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:16:11.053880 7196 log.cc:1196] T 736ed8faceec4f0fa07c62952ea32bf6 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal/wals/736ed8faceec4f0fa07c62952ea32bf6
I20240502 14:16:11.054738 26271 catalog_manager.cc:4830] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): tablet 736ed8faceec4f0fa07c62952ea32bf6 (table rebalancer_extra_table_30 [id=50fafe33ec534b21a087e7ea3eab5a91]) successfully deleted
I20240502 14:16:11.436836 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:11.466028 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:11.521298 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:11.623643 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:16:11.633901 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:11.659149 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240502 14:16:11.678966 26754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.193:37985, user_credentials={real_user=slave}} blocked reactor thread for 102304us
I20240502 14:16:11.707901 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:11.715508 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:11.879834 29175 consensus_queue.cc:237] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:16:11.885542 29307 raft_consensus.cc:1261] T 253449163c65471d8736b92b25da0f92 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:11.885614 26686 raft_consensus.cc:1261] T 253449163c65471d8736b92b25da0f92 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:11.886128 6750 consensus_queue.cc:1035] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:11.886369 7236 consensus_queue.cc:1035] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:11.889227 6750 raft_consensus.cc:2932] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:11.889972 26686 raft_consensus.cc:2932] T 253449163c65471d8736b92b25da0f92 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:11.894407 29309 raft_consensus.cc:2932] T 253449163c65471d8736b92b25da0f92 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:11.898145 26278 catalog_manager.cc:5484] T 253449163c65471d8736b92b25da0f92 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: config changed from index 2 to 3, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) evicted. New cstate: current_term: 1 leader_uuid: "1cb67158133e4648a6eceba32d4dbba3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:11.910619 26538 tablet_service.cc:1514] Processing DeleteTablet for tablet 253449163c65471d8736b92b25da0f92 with delete_type TABLET_DATA_TOMBSTONED (TS ba187e38a2754d60a12de9ac4856430a not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:51696
I20240502 14:16:11.929189 7278 tablet_replica.cc:330] T 253449163c65471d8736b92b25da0f92 P ba187e38a2754d60a12de9ac4856430a: stopping tablet replica
I20240502 14:16:11.929564 7278 raft_consensus.cc:2229] T 253449163c65471d8736b92b25da0f92 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:11.929867 7278 raft_consensus.cc:2258] T 253449163c65471d8736b92b25da0f92 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:11.930491 7278 ts_tablet_manager.cc:1875] T 253449163c65471d8736b92b25da0f92 P ba187e38a2754d60a12de9ac4856430a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:11.936079 7278 ts_tablet_manager.cc:1888] T 253449163c65471d8736b92b25da0f92 P ba187e38a2754d60a12de9ac4856430a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:16:11.936205 7278 log.cc:1196] T 253449163c65471d8736b92b25da0f92 P ba187e38a2754d60a12de9ac4856430a: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/253449163c65471d8736b92b25da0f92
I20240502 14:16:11.937100 26271 catalog_manager.cc:4830] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): tablet 253449163c65471d8736b92b25da0f92 (table rebalancer_extra_table_8 [id=7ae78c7dfcaf4548997196f534416ea8]) successfully deleted
W20240502 14:16:12.169121 26624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.193:37985, user_credentials={real_user=slave}} blocked reactor thread for 82872.3us
I20240502 14:16:12.363392 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:12.408982 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:12.455332 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:12.468323 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:16:12.480572 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:12.525596 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:12.596355 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:12.601275 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:12.784024 29307 consensus_queue.cc:237] T 13136deb9046440f88bf3471ff42312c P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:12.786554 26425 raft_consensus.cc:1261] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer c1100e16b3b9495ca61d4f95e89b6a89: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:12.786645 26558 raft_consensus.cc:1261] T 13136deb9046440f88bf3471ff42312c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer c1100e16b3b9495ca61d4f95e89b6a89: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:12.787281 7185 consensus_queue.cc:1035] T 13136deb9046440f88bf3471ff42312c P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:12.787505 7073 consensus_queue.cc:1035] T 13136deb9046440f88bf3471ff42312c P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:12.790282 7073 raft_consensus.cc:2932] T 13136deb9046440f88bf3471ff42312c P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:12.790704 26425 raft_consensus.cc:2932] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:12.792798 26558 raft_consensus.cc:2932] T 13136deb9046440f88bf3471ff42312c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:12.796831 26276 catalog_manager.cc:5484] T 13136deb9046440f88bf3471ff42312c P c1100e16b3b9495ca61d4f95e89b6a89 reported cstate change: config changed from index 2 to 3, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) evicted. New cstate: current_term: 1 leader_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:12.814144 29155 tablet_service.cc:1514] Processing DeleteTablet for tablet 13136deb9046440f88bf3471ff42312c with delete_type TABLET_DATA_TOMBSTONED (TS 1cb67158133e4648a6eceba32d4dbba3 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:55202
I20240502 14:16:12.814921 7379 tablet_replica.cc:330] T 13136deb9046440f88bf3471ff42312c P 1cb67158133e4648a6eceba32d4dbba3: stopping tablet replica
I20240502 14:16:12.815266 7379 raft_consensus.cc:2229] T 13136deb9046440f88bf3471ff42312c P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:12.815732 7379 raft_consensus.cc:2258] T 13136deb9046440f88bf3471ff42312c P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:12.817317 7379 ts_tablet_manager.cc:1875] T 13136deb9046440f88bf3471ff42312c P 1cb67158133e4648a6eceba32d4dbba3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:12.826429 7379 ts_tablet_manager.cc:1888] T 13136deb9046440f88bf3471ff42312c P 1cb67158133e4648a6eceba32d4dbba3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:16:12.826689 7379 log.cc:1196] T 13136deb9046440f88bf3471ff42312c P 1cb67158133e4648a6eceba32d4dbba3: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal/wals/13136deb9046440f88bf3471ff42312c
I20240502 14:16:12.828258 26271 catalog_manager.cc:4830] TS 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): tablet 13136deb9046440f88bf3471ff42312c (table rebalancer_extra_table_26 [id=88ec6ba556824ae1ac8b611f69608326]) successfully deleted
I20240502 14:16:13.239471 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:13.277410 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:13.286057 26274 maintenance_manager.cc:419] P 09bae9f1528d44c09ade01fa361cebd0: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033397
I20240502 14:16:13.383802 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:13.386659 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:13.428264 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:16:13.431433 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:13.464686 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:13.492893 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:13.613202 26273 maintenance_manager.cc:643] P 09bae9f1528d44c09ade01fa361cebd0: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.326s user 0.084s sys 0.010s Metrics: {"bytes_written":90670,"cfile_init":1,"dirs.queue_time_us":26459,"dirs.run_cpu_time_us":273,"dirs.run_wall_time_us":3741,"drs_written":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1991,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":247,"thread_start_us":36747,"threads_started":1}
I20240502 14:16:13.614739 26274 maintenance_manager.cc:419] P 09bae9f1528d44c09ade01fa361cebd0: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 28808 bytes on disk
I20240502 14:16:13.615897 26273 maintenance_manager.cc:643] P 09bae9f1528d44c09ade01fa361cebd0: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":159,"lbm_reads_lt_1ms":4}
I20240502 14:16:13.683578 26556 consensus_queue.cc:237] T aa4c1d9bf17a48b1b657bb7d03409efb P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:16:13.685801 29176 raft_consensus.cc:1261] T aa4c1d9bf17a48b1b657bb7d03409efb P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:13.686182 26818 raft_consensus.cc:1261] T aa4c1d9bf17a48b1b657bb7d03409efb P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:13.686347 7290 consensus_queue.cc:1035] T aa4c1d9bf17a48b1b657bb7d03409efb P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:13.688647 7290 consensus_queue.cc:1035] T aa4c1d9bf17a48b1b657bb7d03409efb P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:13.689322 7290 raft_consensus.cc:2932] T aa4c1d9bf17a48b1b657bb7d03409efb P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:13.689949 29176 raft_consensus.cc:2932] T aa4c1d9bf17a48b1b657bb7d03409efb P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:13.696907 26278 catalog_manager.cc:5484] T aa4c1d9bf17a48b1b657bb7d03409efb P ba187e38a2754d60a12de9ac4856430a reported cstate change: config changed from index 2 to 3, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) evicted. New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:13.699834 26818 raft_consensus.cc:2932] T aa4c1d9bf17a48b1b657bb7d03409efb P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:13.715662 29554 tablet_service.cc:1514] Processing DeleteTablet for tablet aa4c1d9bf17a48b1b657bb7d03409efb with delete_type TABLET_DATA_TOMBSTONED (TS 4092902b3e674857a0147f1b4f2cdb86 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:36398
I20240502 14:16:13.721496 7483 tablet_replica.cc:330] T aa4c1d9bf17a48b1b657bb7d03409efb P 4092902b3e674857a0147f1b4f2cdb86: stopping tablet replica
I20240502 14:16:13.721832 7483 raft_consensus.cc:2229] T aa4c1d9bf17a48b1b657bb7d03409efb P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:13.722180 7483 raft_consensus.cc:2258] T aa4c1d9bf17a48b1b657bb7d03409efb P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:13.723237 7483 ts_tablet_manager.cc:1875] T aa4c1d9bf17a48b1b657bb7d03409efb P 4092902b3e674857a0147f1b4f2cdb86: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:13.741510 7483 ts_tablet_manager.cc:1888] T aa4c1d9bf17a48b1b657bb7d03409efb P 4092902b3e674857a0147f1b4f2cdb86: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:16:13.741760 7483 log.cc:1196] T aa4c1d9bf17a48b1b657bb7d03409efb P 4092902b3e674857a0147f1b4f2cdb86: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/wal/wals/aa4c1d9bf17a48b1b657bb7d03409efb
I20240502 14:16:13.742709 26272 catalog_manager.cc:4830] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097): tablet aa4c1d9bf17a48b1b657bb7d03409efb (table rebalancer_extra_table_34 [id=b4142465f8a74e2baf1a95d85f3fd781]) successfully deleted
I20240502 14:16:14.029717 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:14.050112 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:14.063130 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:14.086009 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:14.092381 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:16:14.117961 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:14.173167 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:14.177073 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 14:16:14.207063 29511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.199:41781, user_credentials={real_user=slave}} blocked reactor thread for 61346.5us
I20240502 14:16:14.349742 26426 consensus_queue.cc:237] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:16:14.352312 29309 raft_consensus.cc:1261] T e1bdebd770244537a0119baa4e3f8091 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:14.352334 29175 raft_consensus.cc:1261] T e1bdebd770244537a0119baa4e3f8091 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:14.353372 6251 consensus_queue.cc:1035] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:14.354017 6251 consensus_queue.cc:1035] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:14.369545 7563 raft_consensus.cc:2932] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:14.372263 29309 raft_consensus.cc:2932] T e1bdebd770244537a0119baa4e3f8091 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:14.380071 26280 catalog_manager.cc:5484] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:14.381959 29176 raft_consensus.cc:2932] T e1bdebd770244537a0119baa4e3f8091 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:14.416676 29421 tablet_service.cc:1514] Processing DeleteTablet for tablet e1bdebd770244537a0119baa4e3f8091 with delete_type TABLET_DATA_TOMBSTONED (TS 68a3c2b17ed94e06aba66ed4178132ab not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:40652
I20240502 14:16:14.420658 7571 tablet_replica.cc:330] T e1bdebd770244537a0119baa4e3f8091 P 68a3c2b17ed94e06aba66ed4178132ab: stopping tablet replica
I20240502 14:16:14.421495 7571 raft_consensus.cc:2229] T e1bdebd770244537a0119baa4e3f8091 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:14.432670 7571 raft_consensus.cc:2258] T e1bdebd770244537a0119baa4e3f8091 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:14.433456 7571 ts_tablet_manager.cc:1875] T e1bdebd770244537a0119baa4e3f8091 P 68a3c2b17ed94e06aba66ed4178132ab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:14.442260 7571 ts_tablet_manager.cc:1888] T e1bdebd770244537a0119baa4e3f8091 P 68a3c2b17ed94e06aba66ed4178132ab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:16:14.442508 7571 log.cc:1196] T e1bdebd770244537a0119baa4e3f8091 P 68a3c2b17ed94e06aba66ed4178132ab: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal/wals/e1bdebd770244537a0119baa4e3f8091
I20240502 14:16:14.446085 26272 catalog_manager.cc:4830] TS 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): tablet e1bdebd770244537a0119baa4e3f8091 (table rebalancer_extra_table_20 [id=2ccf4597fbc1491f87f24cd5733a46d2]) successfully deleted
I20240502 14:16:14.765239 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:14.772703 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:14.794374 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:14.797570 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:14.830332 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:14.851431 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:16:14.940789 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:14.945984 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:15.129935 26818 consensus_queue.cc:237] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:16:15.133553 29176 raft_consensus.cc:1261] T e45de4b8415446a08e9550f911367624 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:15.133673 29309 raft_consensus.cc:1261] T e45de4b8415446a08e9550f911367624 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:15.134337 7253 consensus_queue.cc:1035] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:15.134788 7540 consensus_queue.cc:1035] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:15.137907 7253 raft_consensus.cc:2932] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:16:15.139763 29309 raft_consensus.cc:2932] T e45de4b8415446a08e9550f911367624 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:16:15.140682 29176 raft_consensus.cc:2932] T e45de4b8415446a08e9550f911367624 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:16:15.147018 26278 catalog_manager.cc:5484] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: config changed from index 2 to 3, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) evicted. New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:16:15.193104 26406 tablet_service.cc:1514] Processing DeleteTablet for tablet e45de4b8415446a08e9550f911367624 with delete_type TABLET_DATA_TOMBSTONED (TS 89af7f423cf44717a08d6b142f0ed49d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:37422
I20240502 14:16:15.212998 7658 tablet_replica.cc:330] T e45de4b8415446a08e9550f911367624 P 89af7f423cf44717a08d6b142f0ed49d: stopping tablet replica
I20240502 14:16:15.213301 7658 raft_consensus.cc:2229] T e45de4b8415446a08e9550f911367624 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:15.213591 7658 raft_consensus.cc:2258] T e45de4b8415446a08e9550f911367624 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:15.214502 7658 ts_tablet_manager.cc:1875] T e45de4b8415446a08e9550f911367624 P 89af7f423cf44717a08d6b142f0ed49d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:15.226516 7658 ts_tablet_manager.cc:1888] T e45de4b8415446a08e9550f911367624 P 89af7f423cf44717a08d6b142f0ed49d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:16:15.226856 7658 log.cc:1196] T e45de4b8415446a08e9550f911367624 P 89af7f423cf44717a08d6b142f0ed49d: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/e45de4b8415446a08e9550f911367624
I20240502 14:16:15.227838 26271 catalog_manager.cc:4830] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): tablet e45de4b8415446a08e9550f911367624 (table rebalancer_extra_table_48 [id=53d47c43f2f14c30b29a5147461aafae]) successfully deleted
I20240502 14:16:15.625777 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:15.666580 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:15.672883 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:15.682740 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:16:15.692579 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:15.693861 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:15.752664 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:15.768469 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:15.916735 29309 consensus_queue.cc:237] T 33a3639c26bf4b35a6e15d6ef173751c P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:16:15.919173 29573 raft_consensus.cc:1261] T 33a3639c26bf4b35a6e15d6ef173751c P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Refusing update from remote peer c1100e16b3b9495ca61d4f95e89b6a89: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:15.919329 26556 raft_consensus.cc:1261] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer c1100e16b3b9495ca61d4f95e89b6a89: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:15.919854 7073 consensus_queue.cc:1035] T 33a3639c26bf4b35a6e15d6ef173751c P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:15.920128 7544 consensus_queue.cc:1035] T 33a3639c26bf4b35a6e15d6ef173751c P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:15.922991 7073 raft_consensus.cc:2932] T 33a3639c26bf4b35a6e15d6ef173751c P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:16:15.923772 29573 raft_consensus.cc:2932] T 33a3639c26bf4b35a6e15d6ef173751c P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:16:15.927606 26556 raft_consensus.cc:2932] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:16:15.931387 26283 catalog_manager.cc:5484] T 33a3639c26bf4b35a6e15d6ef173751c P c1100e16b3b9495ca61d4f95e89b6a89 reported cstate change: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New cstate: current_term: 1 leader_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:16:15.963495 29421 tablet_service.cc:1514] Processing DeleteTablet for tablet 33a3639c26bf4b35a6e15d6ef173751c with delete_type TABLET_DATA_TOMBSTONED (TS 68a3c2b17ed94e06aba66ed4178132ab not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:40652
I20240502 14:16:15.964870 7762 tablet_replica.cc:330] T 33a3639c26bf4b35a6e15d6ef173751c P 68a3c2b17ed94e06aba66ed4178132ab: stopping tablet replica
I20240502 14:16:15.965116 7762 raft_consensus.cc:2229] T 33a3639c26bf4b35a6e15d6ef173751c P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:15.965505 7762 raft_consensus.cc:2258] T 33a3639c26bf4b35a6e15d6ef173751c P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:15.966444 7762 ts_tablet_manager.cc:1875] T 33a3639c26bf4b35a6e15d6ef173751c P 68a3c2b17ed94e06aba66ed4178132ab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:15.974831 7762 ts_tablet_manager.cc:1888] T 33a3639c26bf4b35a6e15d6ef173751c P 68a3c2b17ed94e06aba66ed4178132ab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:16:15.975008 7762 log.cc:1196] T 33a3639c26bf4b35a6e15d6ef173751c P 68a3c2b17ed94e06aba66ed4178132ab: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal/wals/33a3639c26bf4b35a6e15d6ef173751c
I20240502 14:16:15.975920 26272 catalog_manager.cc:4830] TS 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): tablet 33a3639c26bf4b35a6e15d6ef173751c (table rebalancer_extra_table_18 [id=800c8a30773545b3a7670200a5873c0e]) successfully deleted
I20240502 14:16:16.325978 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:16.389627 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:16.396019 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:16.409090 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:16:16.497735 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:16.498440 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:16.500931 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:16.525514 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:16.709937 26425 consensus_queue.cc:237] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:16:16.712020 26686 raft_consensus.cc:1261] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:16.712023 29573 raft_consensus.cc:1261] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:16.712641 7818 consensus_queue.cc:1035] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:16.712996 7563 consensus_queue.cc:1035] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:16.719856 7818 raft_consensus.cc:2932] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:16:16.732770 26686 raft_consensus.cc:2932] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:16:16.739071 29573 raft_consensus.cc:2932] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:16:16.745649 26278 catalog_manager.cc:5484] T d16a118ea4c04a58890c7dd3ff9ec7b4 P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: config changed from index 2 to 3, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:16:16.777830 26798 tablet_service.cc:1514] Processing DeleteTablet for tablet d16a118ea4c04a58890c7dd3ff9ec7b4 with delete_type TABLET_DATA_TOMBSTONED (TS e4cfb8fc5a474177af6848a6efa13a27 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:56010
I20240502 14:16:16.791769 7857 tablet_replica.cc:330] T d16a118ea4c04a58890c7dd3ff9ec7b4 P e4cfb8fc5a474177af6848a6efa13a27: stopping tablet replica
I20240502 14:16:16.792117 7857 raft_consensus.cc:2229] T d16a118ea4c04a58890c7dd3ff9ec7b4 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:16.792371 7857 raft_consensus.cc:2258] T d16a118ea4c04a58890c7dd3ff9ec7b4 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:16.793182 7857 ts_tablet_manager.cc:1875] T d16a118ea4c04a58890c7dd3ff9ec7b4 P e4cfb8fc5a474177af6848a6efa13a27: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:16.800930 7857 ts_tablet_manager.cc:1888] T d16a118ea4c04a58890c7dd3ff9ec7b4 P e4cfb8fc5a474177af6848a6efa13a27: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:16:16.801051 7857 log.cc:1196] T d16a118ea4c04a58890c7dd3ff9ec7b4 P e4cfb8fc5a474177af6848a6efa13a27: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/wals/d16a118ea4c04a58890c7dd3ff9ec7b4
I20240502 14:16:16.802227 26272 catalog_manager.cc:4830] TS e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): tablet d16a118ea4c04a58890c7dd3ff9ec7b4 (table rebalancer_extra_table_54 [id=36399b8bbc064edb98ef3de360e0e9bf]) successfully deleted
I20240502 14:16:17.063771 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:17.104027 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:17.176961 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:16:17.244829 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:17.305864 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:17.317119 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:17.333537 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:17.345796 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:17.558401 29575 consensus_queue.cc:237] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:16:17.560557 26818 raft_consensus.cc:1261] T 83dd34749ff8496483365b30a86db71f P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:17.561105 7630 consensus_queue.cc:1035] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:17.561288 29440 raft_consensus.cc:1261] T 83dd34749ff8496483365b30a86db71f P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:17.561968 7937 consensus_queue.cc:1035] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:17.596675 7749 raft_consensus.cc:2932] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:17.602861 29440 raft_consensus.cc:2932] T 83dd34749ff8496483365b30a86db71f P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:17.604255 26276 catalog_manager.cc:5484] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: config changed from index 2 to 3, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) evicted. New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:17.602370 26818 raft_consensus.cc:2932] T 83dd34749ff8496483365b30a86db71f P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:17.640192 29155 tablet_service.cc:1514] Processing DeleteTablet for tablet 83dd34749ff8496483365b30a86db71f with delete_type TABLET_DATA_TOMBSTONED (TS 1cb67158133e4648a6eceba32d4dbba3 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:55202
I20240502 14:16:17.641861 7954 tablet_replica.cc:330] T 83dd34749ff8496483365b30a86db71f P 1cb67158133e4648a6eceba32d4dbba3: stopping tablet replica
I20240502 14:16:17.642096 7954 raft_consensus.cc:2229] T 83dd34749ff8496483365b30a86db71f P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:17.642300 7954 raft_consensus.cc:2258] T 83dd34749ff8496483365b30a86db71f P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:17.642863 7954 ts_tablet_manager.cc:1875] T 83dd34749ff8496483365b30a86db71f P 1cb67158133e4648a6eceba32d4dbba3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:17.650439 7954 ts_tablet_manager.cc:1888] T 83dd34749ff8496483365b30a86db71f P 1cb67158133e4648a6eceba32d4dbba3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:16:17.652271 7954 log.cc:1196] T 83dd34749ff8496483365b30a86db71f P 1cb67158133e4648a6eceba32d4dbba3: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal/wals/83dd34749ff8496483365b30a86db71f
I20240502 14:16:17.653201 26271 catalog_manager.cc:4830] TS 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): tablet 83dd34749ff8496483365b30a86db71f (table rebalancer_extra_table_8 [id=7ae78c7dfcaf4548997196f534416ea8]) successfully deleted
I20240502 14:16:17.916339 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:17.937953 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:17.965031 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:17.991771 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:16:18.019107 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:18.030917 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:18.066308 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:18.100800 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:18.264302 29309 consensus_queue.cc:237] T 2e59791e95cc452ea30d1ed8bb6178ae P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:18.266625 26425 raft_consensus.cc:1261] T 2e59791e95cc452ea30d1ed8bb6178ae P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer c1100e16b3b9495ca61d4f95e89b6a89: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:18.266671 26686 raft_consensus.cc:1261] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer c1100e16b3b9495ca61d4f95e89b6a89: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:18.267288 7831 consensus_queue.cc:1035] T 2e59791e95cc452ea30d1ed8bb6178ae P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:18.268050 7831 consensus_queue.cc:1035] T 2e59791e95cc452ea30d1ed8bb6178ae P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:18.272765 7544 raft_consensus.cc:2932] T 2e59791e95cc452ea30d1ed8bb6178ae P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:18.273483 26686 raft_consensus.cc:2932] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:18.281170 26280 catalog_manager.cc:5484] T 2e59791e95cc452ea30d1ed8bb6178ae P c1100e16b3b9495ca61d4f95e89b6a89 reported cstate change: config changed from index 2 to 3, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) evicted. New cstate: current_term: 1 leader_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:18.288146 26426 raft_consensus.cc:2932] T 2e59791e95cc452ea30d1ed8bb6178ae P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:18.289418 26538 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e59791e95cc452ea30d1ed8bb6178ae with delete_type TABLET_DATA_TOMBSTONED (TS ba187e38a2754d60a12de9ac4856430a not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:51696
I20240502 14:16:18.290009 8040 tablet_replica.cc:330] T 2e59791e95cc452ea30d1ed8bb6178ae P ba187e38a2754d60a12de9ac4856430a: stopping tablet replica
I20240502 14:16:18.290305 8040 raft_consensus.cc:2229] T 2e59791e95cc452ea30d1ed8bb6178ae P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:18.291814 8040 raft_consensus.cc:2258] T 2e59791e95cc452ea30d1ed8bb6178ae P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:18.294109 8040 ts_tablet_manager.cc:1875] T 2e59791e95cc452ea30d1ed8bb6178ae P ba187e38a2754d60a12de9ac4856430a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:18.299896 8040 ts_tablet_manager.cc:1888] T 2e59791e95cc452ea30d1ed8bb6178ae P ba187e38a2754d60a12de9ac4856430a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:16:18.300097 8040 log.cc:1196] T 2e59791e95cc452ea30d1ed8bb6178ae P ba187e38a2754d60a12de9ac4856430a: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/2e59791e95cc452ea30d1ed8bb6178ae
I20240502 14:16:18.300926 26271 catalog_manager.cc:4830] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): tablet 2e59791e95cc452ea30d1ed8bb6178ae (table rebalancer_extra_table_42 [id=0a0581a07e774016963810a988c20e50]) successfully deleted
I20240502 14:16:18.628322 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:18.667789 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:16:18.697187 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:18.720203 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:18.736992 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:18.800732 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:18.801735 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:18.819620 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:18.966776 26557 consensus_queue.cc:237] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:16:18.969689 29440 raft_consensus.cc:1261] T 4bf6e7410afa4c048f5842ec2a426d77 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:18.969909 26818 raft_consensus.cc:1261] T 4bf6e7410afa4c048f5842ec2a426d77 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:18.970196 7663 consensus_queue.cc:1035] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:18.970592 7416 consensus_queue.cc:1035] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:18.973310 7663 raft_consensus.cc:2932] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:18.973820 29440 raft_consensus.cc:2932] T 4bf6e7410afa4c048f5842ec2a426d77 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:18.973991 26818 raft_consensus.cc:2932] T 4bf6e7410afa4c048f5842ec2a426d77 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:18.983292 26282 catalog_manager.cc:5484] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a reported cstate change: config changed from index 4 to 5, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:19.004056 26669 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bf6e7410afa4c048f5842ec2a426d77 with delete_type TABLET_DATA_TOMBSTONED (TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34150
I20240502 14:16:19.005412 8132 tablet_replica.cc:330] T 4bf6e7410afa4c048f5842ec2a426d77 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: stopping tablet replica
I20240502 14:16:19.005645 8132 raft_consensus.cc:2229] T 4bf6e7410afa4c048f5842ec2a426d77 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:19.005898 8132 raft_consensus.cc:2258] T 4bf6e7410afa4c048f5842ec2a426d77 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:19.006626 8132 ts_tablet_manager.cc:1875] T 4bf6e7410afa4c048f5842ec2a426d77 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:19.012485 8132 ts_tablet_manager.cc:1888] T 4bf6e7410afa4c048f5842ec2a426d77 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:16:19.012713 8132 log.cc:1196] T 4bf6e7410afa4c048f5842ec2a426d77 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/wals/4bf6e7410afa4c048f5842ec2a426d77
I20240502 14:16:19.014600 26272 catalog_manager.cc:4830] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): tablet 4bf6e7410afa4c048f5842ec2a426d77 (table rebalance_test_table_5 [id=8069f2786bf84bf4a012c26e82757ad4]) successfully deleted
I20240502 14:16:19.523023 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:19.537873 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:19.562016 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:19.562422 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:19.580638 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:19.610939 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:19.620895 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:16:19.653664 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:19.815918 26686 consensus_queue.cc:237] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:16:19.818248 29175 raft_consensus.cc:1261] T 451b303a32a3404d8d584203d6704b35 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:19.818248 29440 raft_consensus.cc:1261] T 451b303a32a3404d8d584203d6704b35 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:19.818852 7909 consensus_queue.cc:1035] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:19.819190 7429 consensus_queue.cc:1035] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:19.822883 7429 raft_consensus.cc:2932] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:19.823729 29175 raft_consensus.cc:2932] T 451b303a32a3404d8d584203d6704b35 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:19.828837 29440 raft_consensus.cc:2932] T 451b303a32a3404d8d584203d6704b35 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:19.834544 26278 catalog_manager.cc:5484] T 451b303a32a3404d8d584203d6704b35 P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: config changed from index 2 to 3, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) evicted. New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:19.856321 26538 tablet_service.cc:1514] Processing DeleteTablet for tablet 451b303a32a3404d8d584203d6704b35 with delete_type TABLET_DATA_TOMBSTONED (TS ba187e38a2754d60a12de9ac4856430a not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:51696
I20240502 14:16:19.857017 8225 tablet_replica.cc:330] T 451b303a32a3404d8d584203d6704b35 P ba187e38a2754d60a12de9ac4856430a: stopping tablet replica
I20240502 14:16:19.857221 8225 raft_consensus.cc:2229] T 451b303a32a3404d8d584203d6704b35 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:19.857455 8225 raft_consensus.cc:2258] T 451b303a32a3404d8d584203d6704b35 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:19.858246 8225 ts_tablet_manager.cc:1875] T 451b303a32a3404d8d584203d6704b35 P ba187e38a2754d60a12de9ac4856430a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:19.864698 8225 ts_tablet_manager.cc:1888] T 451b303a32a3404d8d584203d6704b35 P ba187e38a2754d60a12de9ac4856430a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:16:19.864830 8225 log.cc:1196] T 451b303a32a3404d8d584203d6704b35 P ba187e38a2754d60a12de9ac4856430a: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/451b303a32a3404d8d584203d6704b35
I20240502 14:16:19.865564 26271 catalog_manager.cc:4830] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): tablet 451b303a32a3404d8d584203d6704b35 (table rebalancer_extra_table_34 [id=b4142465f8a74e2baf1a95d85f3fd781]) successfully deleted
I20240502 14:16:20.164608 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:20.247880 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:20.278628 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:20.323524 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:20.324602 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:16:20.324852 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:20.334661 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:20.356989 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:20.575762 26557 consensus_queue.cc:237] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:16:20.578130 29306 raft_consensus.cc:1261] T 946239523c0646f1835d7a85b4e7d117 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:20.578282 26818 raft_consensus.cc:1261] T 946239523c0646f1835d7a85b4e7d117 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:20.578744 8123 consensus_queue.cc:1035] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:20.579002 8266 consensus_queue.cc:1035] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:20.581854 8266 raft_consensus.cc:2932] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:20.583031 26818 raft_consensus.cc:2932] T 946239523c0646f1835d7a85b4e7d117 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:20.584005 29306 raft_consensus.cc:2932] T 946239523c0646f1835d7a85b4e7d117 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:20.589995 26280 catalog_manager.cc:5484] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a reported cstate change: config changed from index 4 to 5, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:20.601583 26669 tablet_service.cc:1514] Processing DeleteTablet for tablet 946239523c0646f1835d7a85b4e7d117 with delete_type TABLET_DATA_TOMBSTONED (TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34150
I20240502 14:16:20.602291 8314 tablet_replica.cc:330] T 946239523c0646f1835d7a85b4e7d117 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: stopping tablet replica
I20240502 14:16:20.602689 8314 raft_consensus.cc:2229] T 946239523c0646f1835d7a85b4e7d117 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:20.603087 8314 raft_consensus.cc:2258] T 946239523c0646f1835d7a85b4e7d117 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:20.603940 8314 ts_tablet_manager.cc:1875] T 946239523c0646f1835d7a85b4e7d117 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:20.612706 8314 ts_tablet_manager.cc:1888] T 946239523c0646f1835d7a85b4e7d117 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:16:20.612972 8314 log.cc:1196] T 946239523c0646f1835d7a85b4e7d117 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/wals/946239523c0646f1835d7a85b4e7d117
I20240502 14:16:20.614585 26272 catalog_manager.cc:4830] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): tablet 946239523c0646f1835d7a85b4e7d117 (table rebalance_test_table_1 [id=d7c1bbb69be74fd3a8e809dd8649698d]) successfully deleted
I20240502 14:16:20.933737 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:20.952667 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:20.978680 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:20.989127 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:20.990615 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:20.995479 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:16:21.027006 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:21.043999 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:21.313838 26686 consensus_queue.cc:237] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:16:21.316143 26427 raft_consensus.cc:1261] T 0290ebbcf609427cae8fce6637019336 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:21.316190 29575 raft_consensus.cc:1261] T 0290ebbcf609427cae8fce6637019336 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:21.316704 7909 consensus_queue.cc:1035] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:21.316951 8219 consensus_queue.cc:1035] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:21.319787 7909 raft_consensus.cc:2932] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:16:21.321612 29575 raft_consensus.cc:2932] T 0290ebbcf609427cae8fce6637019336 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:16:21.326434 26426 raft_consensus.cc:2932] T 0290ebbcf609427cae8fce6637019336 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:16:21.330392 26280 catalog_manager.cc:5484] T 0290ebbcf609427cae8fce6637019336 P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: config changed from index 4 to 5, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:16:21.347990 26798 tablet_service.cc:1514] Processing DeleteTablet for tablet 0290ebbcf609427cae8fce6637019336 with delete_type TABLET_DATA_TOMBSTONED (TS e4cfb8fc5a474177af6848a6efa13a27 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:56010
I20240502 14:16:21.348847 8410 tablet_replica.cc:330] T 0290ebbcf609427cae8fce6637019336 P e4cfb8fc5a474177af6848a6efa13a27: stopping tablet replica
I20240502 14:16:21.349109 8410 raft_consensus.cc:2229] T 0290ebbcf609427cae8fce6637019336 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:21.349382 8410 raft_consensus.cc:2258] T 0290ebbcf609427cae8fce6637019336 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:21.350492 8410 ts_tablet_manager.cc:1875] T 0290ebbcf609427cae8fce6637019336 P e4cfb8fc5a474177af6848a6efa13a27: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:21.358239 8410 ts_tablet_manager.cc:1888] T 0290ebbcf609427cae8fce6637019336 P e4cfb8fc5a474177af6848a6efa13a27: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:16:21.358469 8410 log.cc:1196] T 0290ebbcf609427cae8fce6637019336 P e4cfb8fc5a474177af6848a6efa13a27: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/wals/0290ebbcf609427cae8fce6637019336
I20240502 14:16:21.359489 26272 catalog_manager.cc:4830] TS e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): tablet 0290ebbcf609427cae8fce6637019336 (table rebalance_test_table_11 [id=c7861c8f192f462e8ed2198ec9c0c5ea]) successfully deleted
I20240502 14:16:21.446410 29575 consensus_queue.cc:237] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:16:21.448982 29440 raft_consensus.cc:1261] T 01b1a1d502874a3791af520982ecacc0 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:21.449015 29175 raft_consensus.cc:1261] T 01b1a1d502874a3791af520982ecacc0 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:16:21.449635 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): Couldn't send request to peer 69ab78b2a1b74d2bbf2cdeafa0a4af23. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01b1a1d502874a3791af520982ecacc0. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:21.449956 8284 consensus_queue.cc:1035] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:21.455490 8284 consensus_queue.cc:1035] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:21.459295 8180 raft_consensus.cc:2932] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:21.461583 29175 raft_consensus.cc:2932] T 01b1a1d502874a3791af520982ecacc0 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:21.465087 29440 raft_consensus.cc:2932] T 01b1a1d502874a3791af520982ecacc0 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:21.469604 26280 catalog_manager.cc:5484] T 01b1a1d502874a3791af520982ecacc0 P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:21.535017 26815 consensus_queue.cc:237] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:16:21.554992 26687 raft_consensus.cc:1261] T 40aa23be0caa427e89e8fdaf905dc052 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:16:21.555006 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 40aa23be0caa427e89e8fdaf905dc052. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:21.555532 8380 consensus_queue.cc:1035] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:21.556748 26426 raft_consensus.cc:1261] T 40aa23be0caa427e89e8fdaf905dc052 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:21.557387 8380 consensus_queue.cc:1035] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:21.566049 7990 raft_consensus.cc:2932] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:21.566793 26687 raft_consensus.cc:2932] T 40aa23be0caa427e89e8fdaf905dc052 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:21.572415 26426 raft_consensus.cc:2932] T 40aa23be0caa427e89e8fdaf905dc052 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:21.574913 26277 catalog_manager.cc:5484] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: config changed from index -1 to 4, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:22.003398 8499 ts_tablet_manager.cc:902] T 40aa23be0caa427e89e8fdaf905dc052 P c1100e16b3b9495ca61d4f95e89b6a89: Initiating tablet copy from peer e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:16:22.016238 8499 tablet_copy_client.cc:308] T 40aa23be0caa427e89e8fdaf905dc052 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.196:32843
I20240502 14:16:22.017235 26839 tablet_copy_service.cc:140] P e4cfb8fc5a474177af6848a6efa13a27: Received BeginTabletCopySession request for tablet 40aa23be0caa427e89e8fdaf905dc052 from peer c1100e16b3b9495ca61d4f95e89b6a89 ({username='slave'} at 127.19.94.198:38611)
I20240502 14:16:22.017426 26839 tablet_copy_service.cc:161] P e4cfb8fc5a474177af6848a6efa13a27: Beginning new tablet copy session on tablet 40aa23be0caa427e89e8fdaf905dc052 from peer c1100e16b3b9495ca61d4f95e89b6a89 at {username='slave'} at 127.19.94.198:38611: session id = c1100e16b3b9495ca61d4f95e89b6a89-40aa23be0caa427e89e8fdaf905dc052
I20240502 14:16:22.021421 26839 tablet_copy_source_session.cc:205] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:22.037569 8499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40aa23be0caa427e89e8fdaf905dc052. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:22.058640 8506 ts_tablet_manager.cc:902] T 01b1a1d502874a3791af520982ecacc0 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Initiating tablet copy from peer 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:16:22.063755 8499 tablet_copy_client.cc:783] T 40aa23be0caa427e89e8fdaf905dc052 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:22.064148 8499 tablet_copy_client.cc:647] T 40aa23be0caa427e89e8fdaf905dc052 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:22.069690 8499 tablet_copy_client.cc:523] T 40aa23be0caa427e89e8fdaf905dc052 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:22.076931 8499 tablet_bootstrap.cc:492] T 40aa23be0caa427e89e8fdaf905dc052 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:16:22.162797 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:22.173561 8506 tablet_copy_client.cc:308] T 01b1a1d502874a3791af520982ecacc0 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.200:46097
I20240502 14:16:22.175544 29594 tablet_copy_service.cc:140] P 4092902b3e674857a0147f1b4f2cdb86: Received BeginTabletCopySession request for tablet 01b1a1d502874a3791af520982ecacc0 from peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 ({username='slave'} at 127.19.94.195:39709)
I20240502 14:16:22.175806 29594 tablet_copy_service.cc:161] P 4092902b3e674857a0147f1b4f2cdb86: Beginning new tablet copy session on tablet 01b1a1d502874a3791af520982ecacc0 from peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 at {username='slave'} at 127.19.94.195:39709: session id = 69ab78b2a1b74d2bbf2cdeafa0a4af23-01b1a1d502874a3791af520982ecacc0
I20240502 14:16:22.182304 29594 tablet_copy_source_session.cc:205] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:22.184721 8506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01b1a1d502874a3791af520982ecacc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:22.194104 8506 tablet_copy_client.cc:783] T 01b1a1d502874a3791af520982ecacc0 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:22.194597 8506 tablet_copy_client.cc:647] T 01b1a1d502874a3791af520982ecacc0 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:22.215914 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:22.228875 8506 tablet_copy_client.cc:523] T 01b1a1d502874a3791af520982ecacc0 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:22.232964 8506 tablet_bootstrap.cc:492] T 01b1a1d502874a3791af520982ecacc0 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:16:22.254956 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:16:22.275669 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:22.305516 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:22.331913 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:22.349671 8506 tablet_bootstrap.cc:492] T 01b1a1d502874a3791af520982ecacc0 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:22.357510 8499 tablet_bootstrap.cc:492] T 40aa23be0caa427e89e8fdaf905dc052 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:22.358171 8499 tablet_bootstrap.cc:492] T 40aa23be0caa427e89e8fdaf905dc052 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap complete.
I20240502 14:16:22.358513 8499 ts_tablet_manager.cc:1367] T 40aa23be0caa427e89e8fdaf905dc052 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.282s user 0.064s sys 0.010s
I20240502 14:16:22.359313 8499 raft_consensus.cc:348] T 40aa23be0caa427e89e8fdaf905dc052 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:16:22.359719 8499 raft_consensus.cc:729] T 40aa23be0caa427e89e8fdaf905dc052 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:16:22.359997 8499 consensus_queue.cc:260] T 40aa23be0caa427e89e8fdaf905dc052 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:16:22.360810 8499 ts_tablet_manager.cc:1398] T 40aa23be0caa427e89e8fdaf905dc052 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:16:22.361219 8506 tablet_bootstrap.cc:492] T 01b1a1d502874a3791af520982ecacc0 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap complete.
I20240502 14:16:22.361881 26839 tablet_copy_service.cc:342] P e4cfb8fc5a474177af6848a6efa13a27: Request end of tablet copy session c1100e16b3b9495ca61d4f95e89b6a89-40aa23be0caa427e89e8fdaf905dc052 received from {username='slave'} at 127.19.94.198:38611
I20240502 14:16:22.362036 26839 tablet_copy_service.cc:433] P e4cfb8fc5a474177af6848a6efa13a27: ending tablet copy session c1100e16b3b9495ca61d4f95e89b6a89-40aa23be0caa427e89e8fdaf905dc052 on tablet 40aa23be0caa427e89e8fdaf905dc052 with peer c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:16:22.398885 8506 ts_tablet_manager.cc:1367] T 01b1a1d502874a3791af520982ecacc0 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.166s user 0.024s sys 0.007s
I20240502 14:16:22.399865 8506 raft_consensus.cc:348] T 01b1a1d502874a3791af520982ecacc0 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:16:22.400218 8506 raft_consensus.cc:729] T 01b1a1d502874a3791af520982ecacc0 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:16:22.401433 8506 consensus_queue.cc:260] T 01b1a1d502874a3791af520982ecacc0 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:16:22.403815 8506 ts_tablet_manager.cc:1398] T 01b1a1d502874a3791af520982ecacc0 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 14:16:22.406455 29594 tablet_copy_service.cc:342] P 4092902b3e674857a0147f1b4f2cdb86: Request end of tablet copy session 69ab78b2a1b74d2bbf2cdeafa0a4af23-01b1a1d502874a3791af520982ecacc0 received from {username='slave'} at 127.19.94.195:39709
I20240502 14:16:22.406612 29594 tablet_copy_service.cc:433] P 4092902b3e674857a0147f1b4f2cdb86: ending tablet copy session 69ab78b2a1b74d2bbf2cdeafa0a4af23-01b1a1d502874a3791af520982ecacc0 on tablet 01b1a1d502874a3791af520982ecacc0 with peer 69ab78b2a1b74d2bbf2cdeafa0a4af23
I20240502 14:16:22.423804 26686 raft_consensus.cc:1203] T 01b1a1d502874a3791af520982ecacc0 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:16:22.444922 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:22.456552 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:22.540063 29306 raft_consensus.cc:1203] T 40aa23be0caa427e89e8fdaf905dc052 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:16:22.801054 26473 maintenance_manager.cc:419] P 89af7f423cf44717a08d6b142f0ed49d: Scheduling FlushMRSOp(0fc2e6c74dc34a45ae1ec46608642880): perf score=0.033334
I20240502 14:16:22.912371 26735 maintenance_manager.cc:419] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Scheduling FlushMRSOp(65e69adddb7c4dc59e62f616e880cdbc): perf score=0.033365
I20240502 14:16:22.922606 26865 maintenance_manager.cc:419] P e4cfb8fc5a474177af6848a6efa13a27: Scheduling FlushMRSOp(946239523c0646f1835d7a85b4e7d117): perf score=0.033367
I20240502 14:16:22.943866 26364 maintenance_manager.cc:643] P 89af7f423cf44717a08d6b142f0ed49d: FlushMRSOp(0fc2e6c74dc34a45ae1ec46608642880) complete. Timing: real 0.142s user 0.057s sys 0.031s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":430,"dirs.queue_time_us":1219,"dirs.run_cpu_time_us":268,"dirs.run_wall_time_us":1090,"drs_written":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4,"lbm_write_time_us":10307,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":30120,"threads_started":3,"wal-append.queue_time_us":26823}
I20240502 14:16:22.948808 26604 maintenance_manager.cc:419] P ba187e38a2754d60a12de9ac4856430a: Scheduling FlushMRSOp(65e69adddb7c4dc59e62f616e880cdbc): perf score=0.033376
I20240502 14:16:22.949409 26473 maintenance_manager.cc:419] P 89af7f423cf44717a08d6b142f0ed49d: Scheduling UndoDeltaBlockGCOp(0fc2e6c74dc34a45ae1ec46608642880): 4840 bytes on disk
I20240502 14:16:22.956713 26364 maintenance_manager.cc:643] P 89af7f423cf44717a08d6b142f0ed49d: UndoDeltaBlockGCOp(0fc2e6c74dc34a45ae1ec46608642880) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
W20240502 14:16:23.046526 26494 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 55114.4us
I20240502 14:16:23.345523 26495 maintenance_manager.cc:643] P ba187e38a2754d60a12de9ac4856430a: FlushMRSOp(65e69adddb7c4dc59e62f616e880cdbc) complete. Timing: real 0.396s user 0.065s sys 0.020s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":404,"dirs.queue_time_us":26144,"dirs.run_cpu_time_us":228,"dirs.run_wall_time_us":2521,"drs_written":1,"lbm_read_time_us":472,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1097,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":57389,"threads_started":3,"wal-append.queue_time_us":17508}
I20240502 14:16:23.361662 26604 maintenance_manager.cc:419] P ba187e38a2754d60a12de9ac4856430a: Scheduling UndoDeltaBlockGCOp(65e69adddb7c4dc59e62f616e880cdbc): 4585 bytes on disk
I20240502 14:16:23.363876 26495 maintenance_manager.cc:643] P ba187e38a2754d60a12de9ac4856430a: UndoDeltaBlockGCOp(65e69adddb7c4dc59e62f616e880cdbc) complete. Timing: real 0.002s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
I20240502 14:16:23.368358 26604 maintenance_manager.cc:419] P ba187e38a2754d60a12de9ac4856430a: Scheduling FlushMRSOp(0fc2e6c74dc34a45ae1ec46608642880): perf score=0.033488
I20240502 14:16:23.410946 26626 maintenance_manager.cc:643] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: FlushMRSOp(65e69adddb7c4dc59e62f616e880cdbc) complete. Timing: real 0.497s user 0.070s sys 0.013s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":419,"dirs.queue_time_us":18210,"dirs.run_cpu_time_us":255,"dirs.run_wall_time_us":1733,"drs_written":1,"lbm_read_time_us":169,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3017,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":39638,"threads_started":3,"wal-append.queue_time_us":15948}
I20240502 14:16:23.427646 26735 maintenance_manager.cc:419] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Scheduling UndoDeltaBlockGCOp(65e69adddb7c4dc59e62f616e880cdbc): 4585 bytes on disk
I20240502 14:16:23.431761 26626 maintenance_manager.cc:643] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: UndoDeltaBlockGCOp(65e69adddb7c4dc59e62f616e880cdbc) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
I20240502 14:16:23.480912 26756 maintenance_manager.cc:643] P e4cfb8fc5a474177af6848a6efa13a27: FlushMRSOp(946239523c0646f1835d7a85b4e7d117) complete. Timing: real 0.558s user 0.073s sys 0.030s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":41215,"dirs.queue_time_us":83770,"dirs.run_cpu_time_us":244,"dirs.run_wall_time_us":974,"drs_written":1,"lbm_read_time_us":683,"lbm_reads_lt_1ms":4,"lbm_write_time_us":14430,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":169678,"threads_started":3,"wal-append.queue_time_us":38457}
I20240502 14:16:23.506577 26865 maintenance_manager.cc:419] P e4cfb8fc5a474177af6848a6efa13a27: Scheduling FlushMRSOp(65e69adddb7c4dc59e62f616e880cdbc): perf score=0.033526
I20240502 14:16:23.571664 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:23.608911 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:23.651903 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:16:23.681514 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:23.683401 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:23.713265 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:23.734030 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:23.788151 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:23.906888 26756 maintenance_manager.cc:643] P e4cfb8fc5a474177af6848a6efa13a27: FlushMRSOp(65e69adddb7c4dc59e62f616e880cdbc) complete. Timing: real 0.395s user 0.056s sys 0.005s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":35441,"dirs.run_cpu_time_us":240,"dirs.run_wall_time_us":4810,"drs_written":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1050,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":43629,"threads_started":2,"wal-append.queue_time_us":8177}
I20240502 14:16:23.909895 26865 maintenance_manager.cc:419] P e4cfb8fc5a474177af6848a6efa13a27: Scheduling UndoDeltaBlockGCOp(946239523c0646f1835d7a85b4e7d117): 4925 bytes on disk
I20240502 14:16:23.913301 26756 maintenance_manager.cc:643] P e4cfb8fc5a474177af6848a6efa13a27: UndoDeltaBlockGCOp(946239523c0646f1835d7a85b4e7d117) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
I20240502 14:16:23.924456 26865 maintenance_manager.cc:419] P e4cfb8fc5a474177af6848a6efa13a27: Scheduling UndoDeltaBlockGCOp(65e69adddb7c4dc59e62f616e880cdbc): 4585 bytes on disk
I20240502 14:16:23.926798 26756 maintenance_manager.cc:643] P e4cfb8fc5a474177af6848a6efa13a27: UndoDeltaBlockGCOp(65e69adddb7c4dc59e62f616e880cdbc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4}
I20240502 14:16:24.026225 26495 maintenance_manager.cc:643] P ba187e38a2754d60a12de9ac4856430a: FlushMRSOp(0fc2e6c74dc34a45ae1ec46608642880) complete. Timing: real 0.657s user 0.056s sys 0.006s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":47706,"dirs.run_cpu_time_us":323,"dirs.run_wall_time_us":7770,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":15174,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":64718,"threads_started":2,"wal-append.queue_time_us":15263}
I20240502 14:16:24.030905 26604 maintenance_manager.cc:419] P ba187e38a2754d60a12de9ac4856430a: Scheduling FlushMRSOp(946239523c0646f1835d7a85b4e7d117): perf score=0.033670
I20240502 14:16:24.225456 26495 maintenance_manager.cc:643] P ba187e38a2754d60a12de9ac4856430a: FlushMRSOp(946239523c0646f1835d7a85b4e7d117) complete. Timing: real 0.181s user 0.053s sys 0.005s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":51,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":1527,"drs_written":1,"lbm_read_time_us":219,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1056,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":5937,"threads_started":1,"wal-append.queue_time_us":5680}
I20240502 14:16:24.235504 26604 maintenance_manager.cc:419] P ba187e38a2754d60a12de9ac4856430a: Scheduling UndoDeltaBlockGCOp(946239523c0646f1835d7a85b4e7d117): 4925 bytes on disk
I20240502 14:16:24.236369 26495 maintenance_manager.cc:643] P ba187e38a2754d60a12de9ac4856430a: UndoDeltaBlockGCOp(946239523c0646f1835d7a85b4e7d117) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4}
I20240502 14:16:24.239586 26604 maintenance_manager.cc:419] P ba187e38a2754d60a12de9ac4856430a: Scheduling UndoDeltaBlockGCOp(0fc2e6c74dc34a45ae1ec46608642880): 4840 bytes on disk
I20240502 14:16:24.240360 26495 maintenance_manager.cc:643] P ba187e38a2754d60a12de9ac4856430a: UndoDeltaBlockGCOp(0fc2e6c74dc34a45ae1ec46608642880) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":120,"lbm_reads_lt_1ms":4}
I20240502 14:16:24.400928 29575 consensus_queue.cc:237] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:16:24.419812 26687 raft_consensus.cc:1261] T 01b1a1d502874a3791af520982ecacc0 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:24.419996 29438 raft_consensus.cc:1261] T 01b1a1d502874a3791af520982ecacc0 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:24.420778 8284 consensus_queue.cc:1035] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:24.421489 8632 consensus_queue.cc:1035] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:24.449702 8632 raft_consensus.cc:2932] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:24.453203 26687 raft_consensus.cc:2932] T 01b1a1d502874a3791af520982ecacc0 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:24.461792 26277 catalog_manager.cc:5484] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: config changed from index 2 to 3, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) evicted. New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:24.472733 29440 raft_consensus.cc:2932] T 01b1a1d502874a3791af520982ecacc0 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:24.564867 29155 tablet_service.cc:1514] Processing DeleteTablet for tablet 01b1a1d502874a3791af520982ecacc0 with delete_type TABLET_DATA_TOMBSTONED (TS 1cb67158133e4648a6eceba32d4dbba3 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:55202
I20240502 14:16:24.599351 8668 tablet_replica.cc:330] T 01b1a1d502874a3791af520982ecacc0 P 1cb67158133e4648a6eceba32d4dbba3: stopping tablet replica
I20240502 14:16:24.601889 8668 raft_consensus.cc:2229] T 01b1a1d502874a3791af520982ecacc0 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:24.603058 8668 raft_consensus.cc:2258] T 01b1a1d502874a3791af520982ecacc0 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:24.605748 8668 ts_tablet_manager.cc:1875] T 01b1a1d502874a3791af520982ecacc0 P 1cb67158133e4648a6eceba32d4dbba3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:24.646600 8668 ts_tablet_manager.cc:1888] T 01b1a1d502874a3791af520982ecacc0 P 1cb67158133e4648a6eceba32d4dbba3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:16:24.653004 8668 log.cc:1196] T 01b1a1d502874a3791af520982ecacc0 P 1cb67158133e4648a6eceba32d4dbba3: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal/wals/01b1a1d502874a3791af520982ecacc0
I20240502 14:16:24.663872 26271 catalog_manager.cc:4830] TS 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): tablet 01b1a1d502874a3791af520982ecacc0 (table rebalancer_extra_table_26 [id=88ec6ba556824ae1ac8b611f69608326]) successfully deleted
I20240502 14:16:25.192171 26865 maintenance_manager.cc:419] P e4cfb8fc5a474177af6848a6efa13a27: Scheduling FlushMRSOp(39a61744642f40c8ae92da708dc91ca3): perf score=0.033380
I20240502 14:16:25.222430 26735 maintenance_manager.cc:419] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Scheduling FlushMRSOp(66554eb8b44145d4b474707b061c9659): perf score=0.033402
I20240502 14:16:25.238835 26473 maintenance_manager.cc:419] P 89af7f423cf44717a08d6b142f0ed49d: Scheduling FlushMRSOp(39a61744642f40c8ae92da708dc91ca3): perf score=0.033398
I20240502 14:16:25.264359 26604 maintenance_manager.cc:419] P ba187e38a2754d60a12de9ac4856430a: Scheduling FlushMRSOp(66554eb8b44145d4b474707b061c9659): perf score=0.033397
W20240502 14:16:25.447099 26625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.199:41781, user_credentials={real_user=slave}} blocked reactor thread for 51371.3us
I20240502 14:16:25.492453 26626 maintenance_manager.cc:643] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: FlushMRSOp(66554eb8b44145d4b474707b061c9659) complete. Timing: real 0.269s user 0.072s sys 0.019s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":11129,"dirs.run_cpu_time_us":291,"dirs.run_wall_time_us":9360,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":979,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":112287,"threads_started":2,"wal-append.queue_time_us":100863}
I20240502 14:16:25.490769 26364 maintenance_manager.cc:643] P 89af7f423cf44717a08d6b142f0ed49d: FlushMRSOp(39a61744642f40c8ae92da708dc91ca3) complete. Timing: real 0.252s user 0.041s sys 0.040s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":15015,"dirs.run_cpu_time_us":273,"dirs.run_wall_time_us":5495,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2275,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":108209,"threads_started":2,"wal-append.queue_time_us":92845}
I20240502 14:16:25.513818 26473 maintenance_manager.cc:419] P 89af7f423cf44717a08d6b142f0ed49d: Scheduling UndoDeltaBlockGCOp(39a61744642f40c8ae92da708dc91ca3): 4586 bytes on disk
I20240502 14:16:25.515291 26364 maintenance_manager.cc:643] P 89af7f423cf44717a08d6b142f0ed49d: UndoDeltaBlockGCOp(39a61744642f40c8ae92da708dc91ca3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":127,"lbm_reads_lt_1ms":4}
I20240502 14:16:25.569787 26735 maintenance_manager.cc:419] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Scheduling FlushMRSOp(c586354e10cb4bedbbb20bd103ad477e): perf score=0.033468
I20240502 14:16:25.661892 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:25.700773 26495 maintenance_manager.cc:643] P ba187e38a2754d60a12de9ac4856430a: FlushMRSOp(66554eb8b44145d4b474707b061c9659) complete. Timing: real 0.426s user 0.093s sys 0.071s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":14170,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":4472,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":967,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":200350,"threads_started":2,"wal-append.queue_time_us":182084}
I20240502 14:16:25.703544 26604 maintenance_manager.cc:419] P ba187e38a2754d60a12de9ac4856430a: Scheduling UndoDeltaBlockGCOp(66554eb8b44145d4b474707b061c9659): 4925 bytes on disk
I20240502 14:16:25.704376 26495 maintenance_manager.cc:643] P ba187e38a2754d60a12de9ac4856430a: UndoDeltaBlockGCOp(66554eb8b44145d4b474707b061c9659) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
I20240502 14:16:25.752075 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:25.761288 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:16:25.774367 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:25.775230 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:25.763324 26756 maintenance_manager.cc:643] P e4cfb8fc5a474177af6848a6efa13a27: FlushMRSOp(39a61744642f40c8ae92da708dc91ca3) complete. Timing: real 0.571s user 0.063s sys 0.010s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":216231,"compiler_manager_pool.run_cpu_time_us":27,"compiler_manager_pool.run_wall_time_us":27,"dirs.queue_time_us":7473,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":5280,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2934,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":92707,"threads_started":2,"wal-append.queue_time_us":85006}
I20240502 14:16:25.778450 26865 maintenance_manager.cc:419] P e4cfb8fc5a474177af6848a6efa13a27: Scheduling UndoDeltaBlockGCOp(39a61744642f40c8ae92da708dc91ca3): 4586 bytes on disk
I20240502 14:16:25.779088 26756 maintenance_manager.cc:643] P e4cfb8fc5a474177af6848a6efa13a27: UndoDeltaBlockGCOp(39a61744642f40c8ae92da708dc91ca3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
I20240502 14:16:25.790493 26865 maintenance_manager.cc:419] P e4cfb8fc5a474177af6848a6efa13a27: Scheduling FlushMRSOp(c586354e10cb4bedbbb20bd103ad477e): perf score=0.033543
I20240502 14:16:25.795979 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:25.819875 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:25.885458 26626 maintenance_manager.cc:643] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: FlushMRSOp(c586354e10cb4bedbbb20bd103ad477e) complete. Timing: real 0.315s user 0.059s sys 0.016s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":1014,"dirs.run_cpu_time_us":263,"dirs.run_wall_time_us":7930,"drs_written":1,"lbm_read_time_us":118,"lbm_reads_lt_1ms":4,"lbm_write_time_us":994,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":29710,"threads_started":1,"wal-append.queue_time_us":10744}
I20240502 14:16:25.919032 26735 maintenance_manager.cc:419] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Scheduling UndoDeltaBlockGCOp(66554eb8b44145d4b474707b061c9659): 4925 bytes on disk
I20240502 14:16:25.919867 26626 maintenance_manager.cc:643] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: UndoDeltaBlockGCOp(66554eb8b44145d4b474707b061c9659) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":128,"lbm_reads_lt_1ms":4}
I20240502 14:16:25.929452 26735 maintenance_manager.cc:419] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Scheduling UndoDeltaBlockGCOp(c586354e10cb4bedbbb20bd103ad477e): 4840 bytes on disk
I20240502 14:16:25.930188 26626 maintenance_manager.cc:643] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: UndoDeltaBlockGCOp(c586354e10cb4bedbbb20bd103ad477e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
I20240502 14:16:25.966573 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:26.014559 26756 maintenance_manager.cc:643] P e4cfb8fc5a474177af6848a6efa13a27: FlushMRSOp(c586354e10cb4bedbbb20bd103ad477e) complete. Timing: real 0.220s user 0.057s sys 0.020s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":50,"dirs.run_cpu_time_us":160,"dirs.run_wall_time_us":1325,"drs_written":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4,"lbm_write_time_us":888,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":7037,"threads_started":1,"wal-append.queue_time_us":6911}
I20240502 14:16:26.017354 26865 maintenance_manager.cc:419] P e4cfb8fc5a474177af6848a6efa13a27: Scheduling UndoDeltaBlockGCOp(c586354e10cb4bedbbb20bd103ad477e): 4840 bytes on disk
I20240502 14:16:26.020071 26756 maintenance_manager.cc:643] P e4cfb8fc5a474177af6848a6efa13a27: UndoDeltaBlockGCOp(c586354e10cb4bedbbb20bd103ad477e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240502 14:16:26.096657 26816 consensus_queue.cc:237] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:16:26.098688 29309 raft_consensus.cc:1261] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:26.098868 26687 raft_consensus.cc:1261] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:26.099254 8701 consensus_queue.cc:1035] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:26.103354 8701 consensus_queue.cc:1035] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:26.109789 8701 raft_consensus.cc:2932] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:26.115329 29306 raft_consensus.cc:2932] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:26.116005 26687 raft_consensus.cc:2932] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:26.117432 26284 catalog_manager.cc:5484] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: config changed from index 4 to 5, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) evicted. New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:26.155730 26538 tablet_service.cc:1514] Processing DeleteTablet for tablet c9fb1f7eb71c42d2b8f0f6aacd2b0aeb with delete_type TABLET_DATA_TOMBSTONED (TS ba187e38a2754d60a12de9ac4856430a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51696
I20240502 14:16:26.189345 8795 tablet_replica.cc:330] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P ba187e38a2754d60a12de9ac4856430a: stopping tablet replica
I20240502 14:16:26.189718 8795 raft_consensus.cc:2229] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:26.190187 8795 raft_consensus.cc:2258] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:26.191008 8795 ts_tablet_manager.cc:1875] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P ba187e38a2754d60a12de9ac4856430a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:26.218832 8795 ts_tablet_manager.cc:1888] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P ba187e38a2754d60a12de9ac4856430a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:16:26.218997 8795 log.cc:1196] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P ba187e38a2754d60a12de9ac4856430a: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/c9fb1f7eb71c42d2b8f0f6aacd2b0aeb
I20240502 14:16:26.222440 26271 catalog_manager.cc:4830] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): tablet c9fb1f7eb71c42d2b8f0f6aacd2b0aeb (table rebalance_test_table_5 [id=8069f2786bf84bf4a012c26e82757ad4]) successfully deleted
I20240502 14:16:26.518340 26686 consensus_queue.cc:237] T e665b144dfe541dfaf492281d2f72100 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:16:26.522384 26557 raft_consensus.cc:1261] T e665b144dfe541dfaf492281d2f72100 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:26.522814 29440 raft_consensus.cc:1261] T e665b144dfe541dfaf492281d2f72100 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:26.523303 8588 consensus_queue.cc:1035] T e665b144dfe541dfaf492281d2f72100 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:16:26.524030 26625 consensus_peers.cc:485] T e665b144dfe541dfaf492281d2f72100 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): Couldn't send request to peer e4cfb8fc5a474177af6848a6efa13a27. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e665b144dfe541dfaf492281d2f72100. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:26.524202 8588 consensus_queue.cc:1035] T e665b144dfe541dfaf492281d2f72100 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:26.529307 8883 raft_consensus.cc:2932] T e665b144dfe541dfaf492281d2f72100 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:26.529959 29438 raft_consensus.cc:2932] T e665b144dfe541dfaf492281d2f72100 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:26.536294 26556 raft_consensus.cc:2932] T e665b144dfe541dfaf492281d2f72100 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:26.536934 26285 catalog_manager.cc:5484] T e665b144dfe541dfaf492281d2f72100 P 68a3c2b17ed94e06aba66ed4178132ab reported cstate change: config changed from index -1 to 2, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:26.593245 26426 consensus_queue.cc:237] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
W20240502 14:16:26.596407 26362 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b0316714f34d44a5b537cdf20af42ce9. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:26.596524 26556 raft_consensus.cc:1261] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:26.597338 26816 raft_consensus.cc:1261] T b0316714f34d44a5b537cdf20af42ce9 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:26.598104 8882 consensus_queue.cc:1035] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:26.598572 8692 consensus_queue.cc:1035] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:26.601841 8692 raft_consensus.cc:2932] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:26.602465 26556 raft_consensus.cc:2932] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:26.606864 26816 raft_consensus.cc:2932] T b0316714f34d44a5b537cdf20af42ce9 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:26.612344 26281 catalog_manager.cc:5484] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: config changed from index -1 to 2, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:26.681223 26556 consensus_queue.cc:237] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:16:26.683492 29440 raft_consensus.cc:1261] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:26.683720 29174 raft_consensus.cc:1261] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:26.684046 8929 consensus_queue.cc:1035] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:16:26.684329 26493 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5e66bc4ba1f84b50884d6492edb1e0d2. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:26.684623 8929 consensus_queue.cc:1035] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:26.688158 8628 raft_consensus.cc:2932] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:26.688799 29174 raft_consensus.cc:2932] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:26.691784 29440 raft_consensus.cc:2932] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:26.698149 26285 catalog_manager.cc:5484] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 68a3c2b17ed94e06aba66ed4178132ab reported cstate change: config changed from index -1 to 2, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:26.758361 26816 consensus_queue.cc:237] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:16:26.761026 26558 raft_consensus.cc:1261] T 4df2bd4e182c4fe991da385d92d3f76f P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:26.761627 26686 raft_consensus.cc:1261] T 4df2bd4e182c4fe991da385d92d3f76f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:16:26.761842 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4df2bd4e182c4fe991da385d92d3f76f. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:26.762041 8785 consensus_queue.cc:1035] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:26.762318 8701 consensus_queue.cc:1035] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:26.765657 8701 raft_consensus.cc:2932] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:26.766836 26686 raft_consensus.cc:2932] T 4df2bd4e182c4fe991da385d92d3f76f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:26.771203 26558 raft_consensus.cc:2932] T 4df2bd4e182c4fe991da385d92d3f76f P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:26.775357 26281 catalog_manager.cc:5484] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: config changed from index -1 to 4, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:26.816464 26426 consensus_queue.cc:237] T a503853cfec1454f9a5862d1a8e5ead5 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:16:26.819068 29174 raft_consensus.cc:1261] T a503853cfec1454f9a5862d1a8e5ead5 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:26.819152 29440 raft_consensus.cc:1261] T a503853cfec1454f9a5862d1a8e5ead5 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:26.819645 8692 consensus_queue.cc:1035] T a503853cfec1454f9a5862d1a8e5ead5 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:16:26.819962 26363 consensus_peers.cc:485] T a503853cfec1454f9a5862d1a8e5ead5 P 89af7f423cf44717a08d6b142f0ed49d -> Peer e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): Couldn't send request to peer e4cfb8fc5a474177af6848a6efa13a27. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a503853cfec1454f9a5862d1a8e5ead5. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:26.819960 8882 consensus_queue.cc:1035] T a503853cfec1454f9a5862d1a8e5ead5 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:26.822754 8882 raft_consensus.cc:2932] T a503853cfec1454f9a5862d1a8e5ead5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:26.823478 29174 raft_consensus.cc:2932] T a503853cfec1454f9a5862d1a8e5ead5 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:26.827330 29440 raft_consensus.cc:2932] T a503853cfec1454f9a5862d1a8e5ead5 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:26.830253 26284 catalog_manager.cc:5484] T a503853cfec1454f9a5862d1a8e5ead5 P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: config changed from index -1 to 2, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:26.880247 29574 consensus_queue.cc:237] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:26.882637 29306 raft_consensus.cc:1261] T 23f576e1bede478496122a9d1a2e7515 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:26.882833 29440 raft_consensus.cc:1261] T 23f576e1bede478496122a9d1a2e7515 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:16:26.882953 29510 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): Couldn't send request to peer 89af7f423cf44717a08d6b142f0ed49d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 23f576e1bede478496122a9d1a2e7515. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:26.883292 8737 consensus_queue.cc:1035] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:26.883695 8738 consensus_queue.cc:1035] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:26.896633 8737 raft_consensus.cc:2932] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:26.901433 29306 raft_consensus.cc:2932] T 23f576e1bede478496122a9d1a2e7515 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:26.903295 29440 raft_consensus.cc:2932] T 23f576e1bede478496122a9d1a2e7515 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:26.905954 26285 catalog_manager.cc:5484] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: config changed from index -1 to 2, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:26.963551 26816 consensus_queue.cc:237] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:16:26.966818 26686 raft_consensus.cc:1261] T 19f81d8b28564c0dbb0b80bd5018ed5a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:26.966894 29176 raft_consensus.cc:1261] T 19f81d8b28564c0dbb0b80bd5018ed5a P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:26.967501 8701 consensus_queue.cc:1035] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:16:26.967909 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19f81d8b28564c0dbb0b80bd5018ed5a. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:26.968072 8786 consensus_queue.cc:1035] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:26.974061 9009 ts_tablet_manager.cc:902] T b0316714f34d44a5b537cdf20af42ce9 P c1100e16b3b9495ca61d4f95e89b6a89: Initiating tablet copy from peer 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:16:26.973809 8701 raft_consensus.cc:2932] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:26.975481 9009 tablet_copy_client.cc:308] T b0316714f34d44a5b537cdf20af42ce9 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.193:37985
I20240502 14:16:26.976236 26447 tablet_copy_service.cc:140] P 89af7f423cf44717a08d6b142f0ed49d: Received BeginTabletCopySession request for tablet b0316714f34d44a5b537cdf20af42ce9 from peer c1100e16b3b9495ca61d4f95e89b6a89 ({username='slave'} at 127.19.94.198:52561)
I20240502 14:16:26.976464 26447 tablet_copy_service.cc:161] P 89af7f423cf44717a08d6b142f0ed49d: Beginning new tablet copy session on tablet b0316714f34d44a5b537cdf20af42ce9 from peer c1100e16b3b9495ca61d4f95e89b6a89 at {username='slave'} at 127.19.94.198:52561: session id = c1100e16b3b9495ca61d4f95e89b6a89-b0316714f34d44a5b537cdf20af42ce9
I20240502 14:16:26.977520 29176 raft_consensus.cc:2932] T 19f81d8b28564c0dbb0b80bd5018ed5a P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:26.980896 26447 tablet_copy_source_session.cc:205] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:26.983700 9009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0316714f34d44a5b537cdf20af42ce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:26.986711 26686 raft_consensus.cc:2932] T 19f81d8b28564c0dbb0b80bd5018ed5a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:26.989127 9009 tablet_copy_client.cc:783] T b0316714f34d44a5b537cdf20af42ce9 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:26.989347 9009 tablet_copy_client.cc:647] T b0316714f34d44a5b537cdf20af42ce9 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:26.991237 9009 tablet_copy_client.cc:523] T b0316714f34d44a5b537cdf20af42ce9 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:26.993359 26280 catalog_manager.cc:5484] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: config changed from index -1 to 2, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:26.994761 9009 tablet_bootstrap.cc:492] T b0316714f34d44a5b537cdf20af42ce9 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:16:27.006196 9009 tablet_bootstrap.cc:492] T b0316714f34d44a5b537cdf20af42ce9 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:27.006655 9009 tablet_bootstrap.cc:492] T b0316714f34d44a5b537cdf20af42ce9 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap complete.
I20240502 14:16:27.006966 9009 ts_tablet_manager.cc:1367] T b0316714f34d44a5b537cdf20af42ce9 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20240502 14:16:27.008005 9009 raft_consensus.cc:348] T b0316714f34d44a5b537cdf20af42ce9 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:16:27.008500 9009 raft_consensus.cc:729] T b0316714f34d44a5b537cdf20af42ce9 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:16:27.008790 9009 consensus_queue.cc:260] T b0316714f34d44a5b537cdf20af42ce9 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:16:27.009370 9009 ts_tablet_manager.cc:1398] T b0316714f34d44a5b537cdf20af42ce9 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 14:16:27.010134 26447 tablet_copy_service.cc:342] P 89af7f423cf44717a08d6b142f0ed49d: Request end of tablet copy session c1100e16b3b9495ca61d4f95e89b6a89-b0316714f34d44a5b537cdf20af42ce9 received from {username='slave'} at 127.19.94.198:52561
I20240502 14:16:27.010253 26447 tablet_copy_service.cc:433] P 89af7f423cf44717a08d6b142f0ed49d: ending tablet copy session c1100e16b3b9495ca61d4f95e89b6a89-b0316714f34d44a5b537cdf20af42ce9 on tablet b0316714f34d44a5b537cdf20af42ce9 with peer c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:16:27.051581 26687 consensus_queue.cc:237] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:16:27.055214 29440 raft_consensus.cc:1261] T 25b56a948668442da540ca7a198f9fbb P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:27.055512 29176 raft_consensus.cc:1261] T 25b56a948668442da540ca7a198f9fbb P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:16:27.055913 26625 consensus_peers.cc:485] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): Couldn't send request to peer e4cfb8fc5a474177af6848a6efa13a27. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 25b56a948668442da540ca7a198f9fbb. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:27.056080 9032 consensus_queue.cc:1035] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:27.056751 8883 consensus_queue.cc:1035] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:27.068442 8883 raft_consensus.cc:2932] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:27.076277 26280 catalog_manager.cc:5484] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: config changed from index -1 to 2, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:27.074321 29176 raft_consensus.cc:2932] T 25b56a948668442da540ca7a198f9fbb P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:27.081015 9035 ts_tablet_manager.cc:902] T e665b144dfe541dfaf492281d2f72100 P e4cfb8fc5a474177af6848a6efa13a27: Initiating tablet copy from peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:16:27.084719 29440 raft_consensus.cc:2932] T 25b56a948668442da540ca7a198f9fbb P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:27.121840 9035 tablet_copy_client.cc:308] T e665b144dfe541dfaf492281d2f72100 P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.195:44325
I20240502 14:16:27.122627 26709 tablet_copy_service.cc:140] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Received BeginTabletCopySession request for tablet e665b144dfe541dfaf492281d2f72100 from peer e4cfb8fc5a474177af6848a6efa13a27 ({username='slave'} at 127.19.94.196:37053)
I20240502 14:16:27.122784 26709 tablet_copy_service.cc:161] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Beginning new tablet copy session on tablet e665b144dfe541dfaf492281d2f72100 from peer e4cfb8fc5a474177af6848a6efa13a27 at {username='slave'} at 127.19.94.196:37053: session id = e4cfb8fc5a474177af6848a6efa13a27-e665b144dfe541dfaf492281d2f72100
I20240502 14:16:27.125341 26709 tablet_copy_source_session.cc:205] T e665b144dfe541dfaf492281d2f72100 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:27.126787 9035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e665b144dfe541dfaf492281d2f72100. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:27.136426 9035 tablet_copy_client.cc:783] T e665b144dfe541dfaf492281d2f72100 P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:27.136770 9035 tablet_copy_client.cc:647] T e665b144dfe541dfaf492281d2f72100 P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:27.139142 9035 tablet_copy_client.cc:523] T e665b144dfe541dfaf492281d2f72100 P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:27.143929 9035 tablet_bootstrap.cc:492] T e665b144dfe541dfaf492281d2f72100 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:16:27.185319 9035 tablet_bootstrap.cc:492] T e665b144dfe541dfaf492281d2f72100 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:27.185866 9035 tablet_bootstrap.cc:492] T e665b144dfe541dfaf492281d2f72100 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap complete.
I20240502 14:16:27.186144 9035 ts_tablet_manager.cc:1367] T e665b144dfe541dfaf492281d2f72100 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.042s user 0.013s sys 0.025s
I20240502 14:16:27.187319 9035 raft_consensus.cc:348] T e665b144dfe541dfaf492281d2f72100 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:16:27.187546 9035 raft_consensus.cc:729] T e665b144dfe541dfaf492281d2f72100 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:16:27.187798 9035 consensus_queue.cc:260] T e665b144dfe541dfaf492281d2f72100 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:16:27.188505 9035 ts_tablet_manager.cc:1398] T e665b144dfe541dfaf492281d2f72100 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:16:27.189656 26709 tablet_copy_service.cc:342] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Request end of tablet copy session e4cfb8fc5a474177af6848a6efa13a27-e665b144dfe541dfaf492281d2f72100 received from {username='slave'} at 127.19.94.196:37053
I20240502 14:16:27.189796 26709 tablet_copy_service.cc:433] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: ending tablet copy session e4cfb8fc5a474177af6848a6efa13a27-e665b144dfe541dfaf492281d2f72100 on tablet e665b144dfe541dfaf492281d2f72100 with peer e4cfb8fc5a474177af6848a6efa13a27
I20240502 14:16:27.192778 26735 maintenance_manager.cc:419] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Scheduling FlushMRSOp(c9fb1f7eb71c42d2b8f0f6aacd2b0aeb): perf score=0.033351
I20240502 14:16:27.201431 9009 ts_tablet_manager.cc:902] T 5e66bc4ba1f84b50884d6492edb1e0d2 P c1100e16b3b9495ca61d4f95e89b6a89: Initiating tablet copy from peer ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:16:27.223749 26604 maintenance_manager.cc:419] P ba187e38a2754d60a12de9ac4856430a: Scheduling FlushMRSOp(4bf6e7410afa4c048f5842ec2a426d77): perf score=0.033360
I20240502 14:16:27.238873 9009 tablet_copy_client.cc:308] T 5e66bc4ba1f84b50884d6492edb1e0d2 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.194:36025
I20240502 14:16:27.239663 26578 tablet_copy_service.cc:140] P ba187e38a2754d60a12de9ac4856430a: Received BeginTabletCopySession request for tablet 5e66bc4ba1f84b50884d6492edb1e0d2 from peer c1100e16b3b9495ca61d4f95e89b6a89 ({username='slave'} at 127.19.94.198:36021)
I20240502 14:16:27.239877 26578 tablet_copy_service.cc:161] P ba187e38a2754d60a12de9ac4856430a: Beginning new tablet copy session on tablet 5e66bc4ba1f84b50884d6492edb1e0d2 from peer c1100e16b3b9495ca61d4f95e89b6a89 at {username='slave'} at 127.19.94.198:36021: session id = c1100e16b3b9495ca61d4f95e89b6a89-5e66bc4ba1f84b50884d6492edb1e0d2
I20240502 14:16:27.243451 26578 tablet_copy_source_session.cc:205] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:27.245034 9009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e66bc4ba1f84b50884d6492edb1e0d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:27.261788 9009 tablet_copy_client.cc:783] T 5e66bc4ba1f84b50884d6492edb1e0d2 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:27.262742 9009 tablet_copy_client.cc:647] T 5e66bc4ba1f84b50884d6492edb1e0d2 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:27.267927 9009 tablet_copy_client.cc:523] T 5e66bc4ba1f84b50884d6492edb1e0d2 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240502 14:16:27.273427 29245 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 51579us
I20240502 14:16:27.275348 9009 tablet_bootstrap.cc:492] T 5e66bc4ba1f84b50884d6492edb1e0d2 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:16:27.280146 9058 ts_tablet_manager.cc:902] T 4df2bd4e182c4fe991da385d92d3f76f P 68a3c2b17ed94e06aba66ed4178132ab: Initiating tablet copy from peer e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:16:27.283993 26865 maintenance_manager.cc:419] P e4cfb8fc5a474177af6848a6efa13a27: Scheduling FlushMRSOp(c9fb1f7eb71c42d2b8f0f6aacd2b0aeb): perf score=0.033377
I20240502 14:16:27.284904 9059 ts_tablet_manager.cc:902] T 23f576e1bede478496122a9d1a2e7515 P 89af7f423cf44717a08d6b142f0ed49d: Initiating tablet copy from peer 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:16:27.284955 26473 maintenance_manager.cc:419] P 89af7f423cf44717a08d6b142f0ed49d: Scheduling FlushMRSOp(226b35583ee4441a82a66520aeffbb7d): perf score=0.033372
I20240502 14:16:27.303457 9035 ts_tablet_manager.cc:902] T a503853cfec1454f9a5862d1a8e5ead5 P e4cfb8fc5a474177af6848a6efa13a27: Initiating tablet copy from peer 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:16:27.304916 9059 tablet_copy_client.cc:308] T 23f576e1bede478496122a9d1a2e7515 P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.200:46097
I20240502 14:16:27.309479 29594 tablet_copy_service.cc:140] P 4092902b3e674857a0147f1b4f2cdb86: Received BeginTabletCopySession request for tablet 23f576e1bede478496122a9d1a2e7515 from peer 89af7f423cf44717a08d6b142f0ed49d ({username='slave'} at 127.19.94.193:43933)
I20240502 14:16:27.309690 29594 tablet_copy_service.cc:161] P 4092902b3e674857a0147f1b4f2cdb86: Beginning new tablet copy session on tablet 23f576e1bede478496122a9d1a2e7515 from peer 89af7f423cf44717a08d6b142f0ed49d at {username='slave'} at 127.19.94.193:43933: session id = 89af7f423cf44717a08d6b142f0ed49d-23f576e1bede478496122a9d1a2e7515
I20240502 14:16:27.314266 29594 tablet_copy_source_session.cc:205] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:27.317042 9059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23f576e1bede478496122a9d1a2e7515. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:27.348816 9058 tablet_copy_client.cc:308] T 4df2bd4e182c4fe991da385d92d3f76f P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.196:32843
I20240502 14:16:27.351213 26839 tablet_copy_service.cc:140] P e4cfb8fc5a474177af6848a6efa13a27: Received BeginTabletCopySession request for tablet 4df2bd4e182c4fe991da385d92d3f76f from peer 68a3c2b17ed94e06aba66ed4178132ab ({username='slave'} at 127.19.94.199:38699)
I20240502 14:16:27.351557 26839 tablet_copy_service.cc:161] P e4cfb8fc5a474177af6848a6efa13a27: Beginning new tablet copy session on tablet 4df2bd4e182c4fe991da385d92d3f76f from peer 68a3c2b17ed94e06aba66ed4178132ab at {username='slave'} at 127.19.94.199:38699: session id = 68a3c2b17ed94e06aba66ed4178132ab-4df2bd4e182c4fe991da385d92d3f76f
I20240502 14:16:27.357952 26839 tablet_copy_source_session.cc:205] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:27.376874 9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4df2bd4e182c4fe991da385d92d3f76f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:27.387084 9059 tablet_copy_client.cc:783] T 23f576e1bede478496122a9d1a2e7515 P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:27.388578 9059 tablet_copy_client.cc:647] T 23f576e1bede478496122a9d1a2e7515 P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:27.392318 9059 tablet_copy_client.cc:523] T 23f576e1bede478496122a9d1a2e7515 P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:27.394022 9009 tablet_bootstrap.cc:492] T 5e66bc4ba1f84b50884d6492edb1e0d2 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:27.394920 9009 tablet_bootstrap.cc:492] T 5e66bc4ba1f84b50884d6492edb1e0d2 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap complete.
I20240502 14:16:27.397010 9009 ts_tablet_manager.cc:1367] T 5e66bc4ba1f84b50884d6492edb1e0d2 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.122s user 0.009s sys 0.000s
I20240502 14:16:27.397579 9035 tablet_copy_client.cc:308] T a503853cfec1454f9a5862d1a8e5ead5 P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.193:37985
I20240502 14:16:27.402464 26447 tablet_copy_service.cc:140] P 89af7f423cf44717a08d6b142f0ed49d: Received BeginTabletCopySession request for tablet a503853cfec1454f9a5862d1a8e5ead5 from peer e4cfb8fc5a474177af6848a6efa13a27 ({username='slave'} at 127.19.94.196:59299)
I20240502 14:16:27.402657 26447 tablet_copy_service.cc:161] P 89af7f423cf44717a08d6b142f0ed49d: Beginning new tablet copy session on tablet a503853cfec1454f9a5862d1a8e5ead5 from peer e4cfb8fc5a474177af6848a6efa13a27 at {username='slave'} at 127.19.94.196:59299: session id = e4cfb8fc5a474177af6848a6efa13a27-a503853cfec1454f9a5862d1a8e5ead5
I20240502 14:16:27.407092 26447 tablet_copy_source_session.cc:205] T a503853cfec1454f9a5862d1a8e5ead5 P 89af7f423cf44717a08d6b142f0ed49d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:27.400462 9009 raft_consensus.cc:348] T 5e66bc4ba1f84b50884d6492edb1e0d2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:16:27.409065 9035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a503853cfec1454f9a5862d1a8e5ead5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:27.407460 9009 raft_consensus.cc:729] T 5e66bc4ba1f84b50884d6492edb1e0d2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:16:27.417264 9009 consensus_queue.cc:260] T 5e66bc4ba1f84b50884d6492edb1e0d2 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:16:27.419404 9009 ts_tablet_manager.cc:1398] T 5e66bc4ba1f84b50884d6492edb1e0d2 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.022s user 0.000s sys 0.003s
I20240502 14:16:27.425047 26578 tablet_copy_service.cc:342] P ba187e38a2754d60a12de9ac4856430a: Request end of tablet copy session c1100e16b3b9495ca61d4f95e89b6a89-5e66bc4ba1f84b50884d6492edb1e0d2 received from {username='slave'} at 127.19.94.198:36021
I20240502 14:16:27.425196 26578 tablet_copy_service.cc:433] P ba187e38a2754d60a12de9ac4856430a: ending tablet copy session c1100e16b3b9495ca61d4f95e89b6a89-5e66bc4ba1f84b50884d6492edb1e0d2 on tablet 5e66bc4ba1f84b50884d6492edb1e0d2 with peer c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:16:27.433774 9035 tablet_copy_client.cc:783] T a503853cfec1454f9a5862d1a8e5ead5 P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:27.434080 9035 tablet_copy_client.cc:647] T a503853cfec1454f9a5862d1a8e5ead5 P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:27.437618 9059 tablet_bootstrap.cc:492] T 23f576e1bede478496122a9d1a2e7515 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:16:27.438359 9035 tablet_copy_client.cc:523] T a503853cfec1454f9a5862d1a8e5ead5 P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:27.446816 9035 tablet_bootstrap.cc:492] T a503853cfec1454f9a5862d1a8e5ead5 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:16:27.459120 26815 raft_consensus.cc:1203] T e665b144dfe541dfaf492281d2f72100 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:16:27.469838 9058 tablet_copy_client.cc:783] T 4df2bd4e182c4fe991da385d92d3f76f P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:27.472762 9059 tablet_bootstrap.cc:492] T 23f576e1bede478496122a9d1a2e7515 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:27.473227 9059 tablet_bootstrap.cc:492] T 23f576e1bede478496122a9d1a2e7515 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap complete.
I20240502 14:16:27.473501 9059 ts_tablet_manager.cc:1367] T 23f576e1bede478496122a9d1a2e7515 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.036s user 0.006s sys 0.000s
I20240502 14:16:27.474311 9059 raft_consensus.cc:348] T 23f576e1bede478496122a9d1a2e7515 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:27.474579 9059 raft_consensus.cc:729] T 23f576e1bede478496122a9d1a2e7515 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:16:27.474825 9059 consensus_queue.cc:260] T 23f576e1bede478496122a9d1a2e7515 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:27.475425 9059 ts_tablet_manager.cc:1398] T 23f576e1bede478496122a9d1a2e7515 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:16:27.476559 29594 tablet_copy_service.cc:342] P 4092902b3e674857a0147f1b4f2cdb86: Request end of tablet copy session 89af7f423cf44717a08d6b142f0ed49d-23f576e1bede478496122a9d1a2e7515 received from {username='slave'} at 127.19.94.193:43933
I20240502 14:16:27.476703 29594 tablet_copy_service.cc:433] P 4092902b3e674857a0147f1b4f2cdb86: ending tablet copy session 89af7f423cf44717a08d6b142f0ed49d-23f576e1bede478496122a9d1a2e7515 on tablet 23f576e1bede478496122a9d1a2e7515 with peer 89af7f423cf44717a08d6b142f0ed49d
I20240502 14:16:27.477597 9058 tablet_copy_client.cc:647] T 4df2bd4e182c4fe991da385d92d3f76f P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:27.481236 9058 tablet_copy_client.cc:523] T 4df2bd4e182c4fe991da385d92d3f76f P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:27.501291 9083 ts_tablet_manager.cc:902] T 25b56a948668442da540ca7a198f9fbb P e4cfb8fc5a474177af6848a6efa13a27: Initiating tablet copy from peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:16:27.506489 26495 maintenance_manager.cc:643] P ba187e38a2754d60a12de9ac4856430a: FlushMRSOp(4bf6e7410afa4c048f5842ec2a426d77) complete. Timing: real 0.282s user 0.037s sys 0.037s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":86342,"dirs.run_cpu_time_us":207,"dirs.run_wall_time_us":1347,"drs_written":1,"lbm_read_time_us":377,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1039,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":137672,"threads_started":2,"wal-append.queue_time_us":48430}
I20240502 14:16:27.509393 26604 maintenance_manager.cc:419] P ba187e38a2754d60a12de9ac4856430a: Scheduling UndoDeltaBlockGCOp(4bf6e7410afa4c048f5842ec2a426d77): 4586 bytes on disk
I20240502 14:16:27.512181 26626 maintenance_manager.cc:643] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: FlushMRSOp(c9fb1f7eb71c42d2b8f0f6aacd2b0aeb) complete. Timing: real 0.319s user 0.046s sys 0.014s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":30654,"dirs.run_cpu_time_us":264,"dirs.run_wall_time_us":27033,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5826,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":79041,"threads_started":2,"wal-append.queue_time_us":54628}
I20240502 14:16:27.515022 26735 maintenance_manager.cc:419] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Scheduling UndoDeltaBlockGCOp(c9fb1f7eb71c42d2b8f0f6aacd2b0aeb): 4925 bytes on disk
I20240502 14:16:27.517349 26626 maintenance_manager.cc:643] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: UndoDeltaBlockGCOp(c9fb1f7eb71c42d2b8f0f6aacd2b0aeb) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":157,"lbm_reads_lt_1ms":4}
I20240502 14:16:27.520162 26735 maintenance_manager.cc:419] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Scheduling FlushMRSOp(226b35583ee4441a82a66520aeffbb7d): perf score=0.033430
I20240502 14:16:27.521598 26495 maintenance_manager.cc:643] P ba187e38a2754d60a12de9ac4856430a: UndoDeltaBlockGCOp(4bf6e7410afa4c048f5842ec2a426d77) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4}
I20240502 14:16:27.525455 9035 tablet_bootstrap.cc:492] T a503853cfec1454f9a5862d1a8e5ead5 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:27.525967 9035 tablet_bootstrap.cc:492] T a503853cfec1454f9a5862d1a8e5ead5 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap complete.
I20240502 14:16:27.526279 9035 ts_tablet_manager.cc:1367] T a503853cfec1454f9a5862d1a8e5ead5 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.080s user 0.003s sys 0.005s
I20240502 14:16:27.527098 9035 raft_consensus.cc:348] T a503853cfec1454f9a5862d1a8e5ead5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:16:27.527333 9035 raft_consensus.cc:729] T a503853cfec1454f9a5862d1a8e5ead5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:16:27.503222 9058 tablet_bootstrap.cc:492] T 4df2bd4e182c4fe991da385d92d3f76f P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:16:27.527578 9035 consensus_queue.cc:260] T a503853cfec1454f9a5862d1a8e5ead5 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:16:27.520684 9089 ts_tablet_manager.cc:902] T 19f81d8b28564c0dbb0b80bd5018ed5a P 68a3c2b17ed94e06aba66ed4178132ab: Initiating tablet copy from peer e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:16:27.538254 9083 tablet_copy_client.cc:308] T 25b56a948668442da540ca7a198f9fbb P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.195:44325
I20240502 14:16:27.539050 26709 tablet_copy_service.cc:140] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Received BeginTabletCopySession request for tablet 25b56a948668442da540ca7a198f9fbb from peer e4cfb8fc5a474177af6848a6efa13a27 ({username='slave'} at 127.19.94.196:37053)
I20240502 14:16:27.539119 9035 ts_tablet_manager.cc:1398] T a503853cfec1454f9a5862d1a8e5ead5 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.013s user 0.001s sys 0.002s
I20240502 14:16:27.539220 26709 tablet_copy_service.cc:161] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Beginning new tablet copy session on tablet 25b56a948668442da540ca7a198f9fbb from peer e4cfb8fc5a474177af6848a6efa13a27 at {username='slave'} at 127.19.94.196:37053: session id = e4cfb8fc5a474177af6848a6efa13a27-25b56a948668442da540ca7a198f9fbb
I20240502 14:16:27.539856 26447 tablet_copy_service.cc:342] P 89af7f423cf44717a08d6b142f0ed49d: Request end of tablet copy session e4cfb8fc5a474177af6848a6efa13a27-a503853cfec1454f9a5862d1a8e5ead5 received from {username='slave'} at 127.19.94.196:59299
I20240502 14:16:27.539988 26447 tablet_copy_service.cc:433] P 89af7f423cf44717a08d6b142f0ed49d: ending tablet copy session e4cfb8fc5a474177af6848a6efa13a27-a503853cfec1454f9a5862d1a8e5ead5 on tablet a503853cfec1454f9a5862d1a8e5ead5 with peer e4cfb8fc5a474177af6848a6efa13a27
I20240502 14:16:27.547245 26709 tablet_copy_source_session.cc:205] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:27.552073 9083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25b56a948668442da540ca7a198f9fbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:27.568656 29309 raft_consensus.cc:1203] T b0316714f34d44a5b537cdf20af42ce9 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:16:27.582767 9089 tablet_copy_client.cc:308] T 19f81d8b28564c0dbb0b80bd5018ed5a P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.196:32843
I20240502 14:16:27.583781 26839 tablet_copy_service.cc:140] P e4cfb8fc5a474177af6848a6efa13a27: Received BeginTabletCopySession request for tablet 19f81d8b28564c0dbb0b80bd5018ed5a from peer 68a3c2b17ed94e06aba66ed4178132ab ({username='slave'} at 127.19.94.199:38699)
I20240502 14:16:27.583947 26839 tablet_copy_service.cc:161] P e4cfb8fc5a474177af6848a6efa13a27: Beginning new tablet copy session on tablet 19f81d8b28564c0dbb0b80bd5018ed5a from peer 68a3c2b17ed94e06aba66ed4178132ab at {username='slave'} at 127.19.94.199:38699: session id = 68a3c2b17ed94e06aba66ed4178132ab-19f81d8b28564c0dbb0b80bd5018ed5a
I20240502 14:16:27.586289 9083 tablet_copy_client.cc:783] T 25b56a948668442da540ca7a198f9fbb P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:27.586556 9083 tablet_copy_client.cc:647] T 25b56a948668442da540ca7a198f9fbb P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:27.587724 26839 tablet_copy_source_session.cc:205] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:27.591084 9089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19f81d8b28564c0dbb0b80bd5018ed5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:27.592486 9083 tablet_copy_client.cc:523] T 25b56a948668442da540ca7a198f9fbb P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:27.603116 29309 raft_consensus.cc:1203] T 5e66bc4ba1f84b50884d6492edb1e0d2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:16:27.613874 9083 tablet_bootstrap.cc:492] T 25b56a948668442da540ca7a198f9fbb P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
W20240502 14:16:27.620719 26754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.194:36025, user_credentials={real_user=slave}} blocked reactor thread for 50860.3us
I20240502 14:16:27.629295 26364 maintenance_manager.cc:643] P 89af7f423cf44717a08d6b142f0ed49d: FlushMRSOp(226b35583ee4441a82a66520aeffbb7d) complete. Timing: real 0.344s user 0.035s sys 0.018s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":8812,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":3666,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1000,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":9774,"threads_started":2,"wal-append.queue_time_us":679}
I20240502 14:16:27.632707 26473 maintenance_manager.cc:419] P 89af7f423cf44717a08d6b142f0ed49d: Scheduling UndoDeltaBlockGCOp(226b35583ee4441a82a66520aeffbb7d): 4840 bytes on disk
I20240502 14:16:27.637059 26364 maintenance_manager.cc:643] P 89af7f423cf44717a08d6b142f0ed49d: UndoDeltaBlockGCOp(226b35583ee4441a82a66520aeffbb7d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
I20240502 14:16:27.640698 9089 tablet_copy_client.cc:783] T 19f81d8b28564c0dbb0b80bd5018ed5a P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:27.641008 9089 tablet_copy_client.cc:647] T 19f81d8b28564c0dbb0b80bd5018ed5a P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:27.643420 9089 tablet_copy_client.cc:523] T 19f81d8b28564c0dbb0b80bd5018ed5a P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:27.644098 26425 raft_consensus.cc:1203] T 23f576e1bede478496122a9d1a2e7515 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:16:27.661849 9083 tablet_bootstrap.cc:492] T 25b56a948668442da540ca7a198f9fbb P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:27.662436 9083 tablet_bootstrap.cc:492] T 25b56a948668442da540ca7a198f9fbb P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap complete.
I20240502 14:16:27.662621 9089 tablet_bootstrap.cc:492] T 19f81d8b28564c0dbb0b80bd5018ed5a P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:16:27.663476 9083 ts_tablet_manager.cc:1367] T 25b56a948668442da540ca7a198f9fbb P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.050s user 0.005s sys 0.005s
I20240502 14:16:27.667692 9083 raft_consensus.cc:348] T 25b56a948668442da540ca7a198f9fbb P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:16:27.673869 9083 raft_consensus.cc:729] T 25b56a948668442da540ca7a198f9fbb P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:16:27.674623 9083 consensus_queue.cc:260] T 25b56a948668442da540ca7a198f9fbb P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:16:27.676100 9083 ts_tablet_manager.cc:1398] T 25b56a948668442da540ca7a198f9fbb P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240502 14:16:27.678249 26709 tablet_copy_service.cc:342] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Request end of tablet copy session e4cfb8fc5a474177af6848a6efa13a27-25b56a948668442da540ca7a198f9fbb received from {username='slave'} at 127.19.94.196:37053
I20240502 14:16:27.678377 26709 tablet_copy_service.cc:433] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: ending tablet copy session e4cfb8fc5a474177af6848a6efa13a27-25b56a948668442da540ca7a198f9fbb on tablet 25b56a948668442da540ca7a198f9fbb with peer e4cfb8fc5a474177af6848a6efa13a27
I20240502 14:16:27.703716 26756 maintenance_manager.cc:643] P e4cfb8fc5a474177af6848a6efa13a27: FlushMRSOp(c9fb1f7eb71c42d2b8f0f6aacd2b0aeb) complete. Timing: real 0.419s user 0.064s sys 0.011s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":89292,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":3152,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3212,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":105161,"threads_started":2,"wal-append.queue_time_us":15583}
I20240502 14:16:27.706918 26865 maintenance_manager.cc:419] P e4cfb8fc5a474177af6848a6efa13a27: Scheduling FlushMRSOp(4bf6e7410afa4c048f5842ec2a426d77): perf score=0.033483
I20240502 14:16:27.708150 26815 raft_consensus.cc:1203] T a503853cfec1454f9a5862d1a8e5ead5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:16:27.714993 9089 tablet_bootstrap.cc:492] T 19f81d8b28564c0dbb0b80bd5018ed5a P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:27.715461 9089 tablet_bootstrap.cc:492] T 19f81d8b28564c0dbb0b80bd5018ed5a P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap complete.
I20240502 14:16:27.715727 9089 ts_tablet_manager.cc:1367] T 19f81d8b28564c0dbb0b80bd5018ed5a P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.053s user 0.012s sys 0.000s
I20240502 14:16:27.716521 9089 raft_consensus.cc:348] T 19f81d8b28564c0dbb0b80bd5018ed5a P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:16:27.716758 9089 raft_consensus.cc:729] T 19f81d8b28564c0dbb0b80bd5018ed5a P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:16:27.717003 9089 consensus_queue.cc:260] T 19f81d8b28564c0dbb0b80bd5018ed5a P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:16:27.717651 9089 ts_tablet_manager.cc:1398] T 19f81d8b28564c0dbb0b80bd5018ed5a P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 14:16:27.725082 26839 tablet_copy_service.cc:342] P e4cfb8fc5a474177af6848a6efa13a27: Request end of tablet copy session 68a3c2b17ed94e06aba66ed4178132ab-19f81d8b28564c0dbb0b80bd5018ed5a received from {username='slave'} at 127.19.94.199:38699
I20240502 14:16:27.725252 26839 tablet_copy_service.cc:433] P e4cfb8fc5a474177af6848a6efa13a27: ending tablet copy session 68a3c2b17ed94e06aba66ed4178132ab-19f81d8b28564c0dbb0b80bd5018ed5a on tablet 19f81d8b28564c0dbb0b80bd5018ed5a with peer 68a3c2b17ed94e06aba66ed4178132ab
I20240502 14:16:27.748459 9058 tablet_bootstrap.cc:492] T 4df2bd4e182c4fe991da385d92d3f76f P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:27.758529 9058 tablet_bootstrap.cc:492] T 4df2bd4e182c4fe991da385d92d3f76f P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap complete.
I20240502 14:16:27.758921 9058 ts_tablet_manager.cc:1367] T 4df2bd4e182c4fe991da385d92d3f76f P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.256s user 0.048s sys 0.004s
I20240502 14:16:27.759987 9058 raft_consensus.cc:348] T 4df2bd4e182c4fe991da385d92d3f76f P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:16:27.760273 9058 raft_consensus.cc:729] T 4df2bd4e182c4fe991da385d92d3f76f P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:16:27.760640 9058 consensus_queue.cc:260] T 4df2bd4e182c4fe991da385d92d3f76f P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:16:27.761409 9058 ts_tablet_manager.cc:1398] T 4df2bd4e182c4fe991da385d92d3f76f P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:16:27.766263 26839 tablet_copy_service.cc:342] P e4cfb8fc5a474177af6848a6efa13a27: Request end of tablet copy session 68a3c2b17ed94e06aba66ed4178132ab-4df2bd4e182c4fe991da385d92d3f76f received from {username='slave'} at 127.19.94.199:38699
I20240502 14:16:27.766423 26839 tablet_copy_service.cc:433] P e4cfb8fc5a474177af6848a6efa13a27: ending tablet copy session 68a3c2b17ed94e06aba66ed4178132ab-4df2bd4e182c4fe991da385d92d3f76f on tablet 4df2bd4e182c4fe991da385d92d3f76f with peer 68a3c2b17ed94e06aba66ed4178132ab
I20240502 14:16:27.782863 29440 raft_consensus.cc:1203] T 4df2bd4e182c4fe991da385d92d3f76f P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:16:27.810933 26626 maintenance_manager.cc:643] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: FlushMRSOp(226b35583ee4441a82a66520aeffbb7d) complete. Timing: real 0.289s user 0.049s sys 0.029s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":51,"dirs.run_cpu_time_us":177,"dirs.run_wall_time_us":2113,"drs_written":1,"lbm_read_time_us":162,"lbm_reads_lt_1ms":4,"lbm_write_time_us":814,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":8494,"threads_started":1,"wal-append.queue_time_us":8384}
I20240502 14:16:27.815032 26735 maintenance_manager.cc:419] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Scheduling UndoDeltaBlockGCOp(226b35583ee4441a82a66520aeffbb7d): 4840 bytes on disk
I20240502 14:16:27.815904 26626 maintenance_manager.cc:643] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: UndoDeltaBlockGCOp(226b35583ee4441a82a66520aeffbb7d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240502 14:16:27.923355 26815 raft_consensus.cc:1203] T 25b56a948668442da540ca7a198f9fbb P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:16:27.932616 29440 raft_consensus.cc:1203] T 19f81d8b28564c0dbb0b80bd5018ed5a P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:16:27.944811 26756 maintenance_manager.cc:643] P e4cfb8fc5a474177af6848a6efa13a27: FlushMRSOp(4bf6e7410afa4c048f5842ec2a426d77) complete. Timing: real 0.231s user 0.058s sys 0.032s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":45,"dirs.run_cpu_time_us":160,"dirs.run_wall_time_us":1755,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1466,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":26333,"threads_started":1,"wal-append.queue_time_us":26442}
I20240502 14:16:27.947206 26865 maintenance_manager.cc:419] P e4cfb8fc5a474177af6848a6efa13a27: Scheduling UndoDeltaBlockGCOp(c9fb1f7eb71c42d2b8f0f6aacd2b0aeb): 4925 bytes on disk
I20240502 14:16:27.948011 26756 maintenance_manager.cc:643] P e4cfb8fc5a474177af6848a6efa13a27: UndoDeltaBlockGCOp(c9fb1f7eb71c42d2b8f0f6aacd2b0aeb) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":244,"lbm_reads_lt_1ms":4}
I20240502 14:16:27.950933 26865 maintenance_manager.cc:419] P e4cfb8fc5a474177af6848a6efa13a27: Scheduling UndoDeltaBlockGCOp(4bf6e7410afa4c048f5842ec2a426d77): 4586 bytes on disk
I20240502 14:16:27.951613 26756 maintenance_manager.cc:643] P e4cfb8fc5a474177af6848a6efa13a27: UndoDeltaBlockGCOp(4bf6e7410afa4c048f5842ec2a426d77) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4}
I20240502 14:16:28.000059 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:28.017971 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:28.018589 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:28.022369 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:28.073359 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:28.095209 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:16:28.095361 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:28.124161 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:28.333679 26686 consensus_queue.cc:237] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:16:28.336418 26815 raft_consensus.cc:1261] T 25b56a948668442da540ca7a198f9fbb P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:28.336458 29174 raft_consensus.cc:1261] T 25b56a948668442da540ca7a198f9fbb P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:28.337064 8883 consensus_queue.cc:1035] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:28.337702 9036 consensus_queue.cc:1035] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:28.340862 9036 raft_consensus.cc:2932] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:28.342986 29176 raft_consensus.cc:2932] T 25b56a948668442da540ca7a198f9fbb P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:28.346900 26815 raft_consensus.cc:2932] T 25b56a948668442da540ca7a198f9fbb P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:28.350831 26285 catalog_manager.cc:5484] T 25b56a948668442da540ca7a198f9fbb P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:28.362807 29421 tablet_service.cc:1514] Processing DeleteTablet for tablet 25b56a948668442da540ca7a198f9fbb with delete_type TABLET_DATA_TOMBSTONED (TS 68a3c2b17ed94e06aba66ed4178132ab not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:40652
I20240502 14:16:28.365343 9190 tablet_replica.cc:330] T 25b56a948668442da540ca7a198f9fbb P 68a3c2b17ed94e06aba66ed4178132ab: stopping tablet replica
I20240502 14:16:28.365600 9190 raft_consensus.cc:2229] T 25b56a948668442da540ca7a198f9fbb P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:28.365844 9190 raft_consensus.cc:2258] T 25b56a948668442da540ca7a198f9fbb P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:28.366612 9190 ts_tablet_manager.cc:1875] T 25b56a948668442da540ca7a198f9fbb P 68a3c2b17ed94e06aba66ed4178132ab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:28.373396 9190 ts_tablet_manager.cc:1888] T 25b56a948668442da540ca7a198f9fbb P 68a3c2b17ed94e06aba66ed4178132ab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:16:28.373535 9190 log.cc:1196] T 25b56a948668442da540ca7a198f9fbb P 68a3c2b17ed94e06aba66ed4178132ab: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal/wals/25b56a948668442da540ca7a198f9fbb
I20240502 14:16:28.374378 26272 catalog_manager.cc:4830] TS 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): tablet 25b56a948668442da540ca7a198f9fbb (table rebalancer_extra_table_32 [id=eb9f13fa5bdc4cc3b3c074bc196dfb8c]) successfully deleted
I20240502 14:16:28.743211 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:28.747975 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:16:28.764108 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:28.764499 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:28.765646 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:28.774906 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:28.820122 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:28.861667 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:29.000360 26816 consensus_queue.cc:237] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:16:29.002869 29174 raft_consensus.cc:1261] T 19f81d8b28564c0dbb0b80bd5018ed5a P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:29.003427 29440 raft_consensus.cc:1261] T 19f81d8b28564c0dbb0b80bd5018ed5a P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:29.003651 9212 consensus_queue.cc:1035] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:29.004180 8786 consensus_queue.cc:1035] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:29.007205 9212 raft_consensus.cc:2932] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:29.008455 29440 raft_consensus.cc:2932] T 19f81d8b28564c0dbb0b80bd5018ed5a P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:29.008733 29174 raft_consensus.cc:2932] T 19f81d8b28564c0dbb0b80bd5018ed5a P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:29.014149 26281 catalog_manager.cc:5484] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: config changed from index 2 to 3, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:29.044452 26669 tablet_service.cc:1514] Processing DeleteTablet for tablet 19f81d8b28564c0dbb0b80bd5018ed5a with delete_type TABLET_DATA_TOMBSTONED (TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:34150
I20240502 14:16:29.050863 9280 tablet_replica.cc:330] T 19f81d8b28564c0dbb0b80bd5018ed5a P 69ab78b2a1b74d2bbf2cdeafa0a4af23: stopping tablet replica
I20240502 14:16:29.054714 9280 raft_consensus.cc:2229] T 19f81d8b28564c0dbb0b80bd5018ed5a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:29.055088 9280 raft_consensus.cc:2258] T 19f81d8b28564c0dbb0b80bd5018ed5a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:29.055989 9280 ts_tablet_manager.cc:1875] T 19f81d8b28564c0dbb0b80bd5018ed5a P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:29.072180 9280 ts_tablet_manager.cc:1888] T 19f81d8b28564c0dbb0b80bd5018ed5a P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:16:29.072377 9280 log.cc:1196] T 19f81d8b28564c0dbb0b80bd5018ed5a P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/wals/19f81d8b28564c0dbb0b80bd5018ed5a
I20240502 14:16:29.081456 26272 catalog_manager.cc:4830] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): tablet 19f81d8b28564c0dbb0b80bd5018ed5a (table rebalancer_extra_table_42 [id=0a0581a07e774016963810a988c20e50]) successfully deleted
I20240502 14:16:29.331749 26735 maintenance_manager.cc:419] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Scheduling FlushMRSOp(aedd72fd3aeb48d68f26a042069cebab): perf score=0.033343
I20240502 14:16:29.412137 26473 maintenance_manager.cc:419] P 89af7f423cf44717a08d6b142f0ed49d: Scheduling FlushMRSOp(9f3cbedfa1da42698615a544d9dc3276): perf score=0.033377
I20240502 14:16:29.452812 26626 maintenance_manager.cc:643] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: FlushMRSOp(aedd72fd3aeb48d68f26a042069cebab) complete. Timing: real 0.120s user 0.029s sys 0.027s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":6640,"dirs.run_cpu_time_us":207,"dirs.run_wall_time_us":7849,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1185,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":40859,"threads_started":2,"wal-append.queue_time_us":34187}
I20240502 14:16:29.455547 26735 maintenance_manager.cc:419] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Scheduling UndoDeltaBlockGCOp(aedd72fd3aeb48d68f26a042069cebab): 4586 bytes on disk
I20240502 14:16:29.460675 26626 maintenance_manager.cc:643] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: UndoDeltaBlockGCOp(aedd72fd3aeb48d68f26a042069cebab) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":701,"lbm_reads_lt_1ms":4}
I20240502 14:16:29.470960 26865 maintenance_manager.cc:419] P e4cfb8fc5a474177af6848a6efa13a27: Scheduling FlushMRSOp(d7f02f897ce54fefb03832da3c2d95e0): perf score=0.033388
I20240502 14:16:29.544461 26604 maintenance_manager.cc:419] P ba187e38a2754d60a12de9ac4856430a: Scheduling FlushMRSOp(9f3cbedfa1da42698615a544d9dc3276): perf score=0.033402
I20240502 14:16:29.597002 26364 maintenance_manager.cc:643] P 89af7f423cf44717a08d6b142f0ed49d: FlushMRSOp(9f3cbedfa1da42698615a544d9dc3276) complete. Timing: real 0.184s user 0.054s sys 0.026s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":6685,"dirs.run_cpu_time_us":243,"dirs.run_wall_time_us":4658,"drs_written":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4,"lbm_write_time_us":905,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":19517,"threads_started":2,"wal-append.queue_time_us":12805}
I20240502 14:16:29.601619 26473 maintenance_manager.cc:419] P 89af7f423cf44717a08d6b142f0ed49d: Scheduling UndoDeltaBlockGCOp(9f3cbedfa1da42698615a544d9dc3276): 4925 bytes on disk
I20240502 14:16:29.604204 26364 maintenance_manager.cc:643] P 89af7f423cf44717a08d6b142f0ed49d: UndoDeltaBlockGCOp(9f3cbedfa1da42698615a544d9dc3276) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":155,"lbm_reads_lt_1ms":4}
I20240502 14:16:29.608278 26473 maintenance_manager.cc:419] P 89af7f423cf44717a08d6b142f0ed49d: Scheduling FlushMRSOp(aedd72fd3aeb48d68f26a042069cebab): perf score=0.033425
I20240502 14:16:29.635697 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:29.702891 26364 maintenance_manager.cc:643] P 89af7f423cf44717a08d6b142f0ed49d: FlushMRSOp(aedd72fd3aeb48d68f26a042069cebab) complete. Timing: real 0.094s user 0.045s sys 0.019s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":61,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":1548,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2070,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":27270,"threads_started":1,"wal-append.queue_time_us":25551}
I20240502 14:16:29.706290 26473 maintenance_manager.cc:419] P 89af7f423cf44717a08d6b142f0ed49d: Scheduling UndoDeltaBlockGCOp(aedd72fd3aeb48d68f26a042069cebab): 4586 bytes on disk
I20240502 14:16:29.706943 26364 maintenance_manager.cc:643] P 89af7f423cf44717a08d6b142f0ed49d: UndoDeltaBlockGCOp(aedd72fd3aeb48d68f26a042069cebab) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
I20240502 14:16:29.748328 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:29.765306 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:29.769543 26756 maintenance_manager.cc:643] P e4cfb8fc5a474177af6848a6efa13a27: FlushMRSOp(d7f02f897ce54fefb03832da3c2d95e0) complete. Timing: real 0.298s user 0.074s sys 0.010s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":16922,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":5138,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":964,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":34779,"threads_started":2,"wal-append.queue_time_us":24838}
I20240502 14:16:29.772915 26865 maintenance_manager.cc:419] P e4cfb8fc5a474177af6848a6efa13a27: Scheduling UndoDeltaBlockGCOp(d7f02f897ce54fefb03832da3c2d95e0): 4840 bytes on disk
I20240502 14:16:29.775832 26756 maintenance_manager.cc:643] P e4cfb8fc5a474177af6848a6efa13a27: UndoDeltaBlockGCOp(d7f02f897ce54fefb03832da3c2d95e0) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240502 14:16:29.780711 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:16:29.793749 26495 maintenance_manager.cc:643] P ba187e38a2754d60a12de9ac4856430a: FlushMRSOp(9f3cbedfa1da42698615a544d9dc3276) complete. Timing: real 0.249s user 0.049s sys 0.019s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":17149,"dirs.run_cpu_time_us":227,"dirs.run_wall_time_us":3299,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1006,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":47593,"threads_started":2,"wal-append.queue_time_us":30455}
I20240502 14:16:29.797715 26604 maintenance_manager.cc:419] P ba187e38a2754d60a12de9ac4856430a: Scheduling FlushMRSOp(d7f02f897ce54fefb03832da3c2d95e0): perf score=0.033467
I20240502 14:16:29.801609 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:29.815088 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:29.823462 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:29.849138 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240502 14:16:29.892551 29244 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.194:36025, user_credentials={real_user=slave}} blocked reactor thread for 51409.4us
I20240502 14:16:29.915334 26495 maintenance_manager.cc:643] P ba187e38a2754d60a12de9ac4856430a: FlushMRSOp(d7f02f897ce54fefb03832da3c2d95e0) complete. Timing: real 0.112s user 0.035s sys 0.020s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":49,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":1606,"drs_written":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1357,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":4761,"threads_started":1,"wal-append.queue_time_us":4903}
I20240502 14:16:29.918267 26604 maintenance_manager.cc:419] P ba187e38a2754d60a12de9ac4856430a: Scheduling UndoDeltaBlockGCOp(9f3cbedfa1da42698615a544d9dc3276): 4925 bytes on disk
I20240502 14:16:29.919064 26495 maintenance_manager.cc:643] P ba187e38a2754d60a12de9ac4856430a: UndoDeltaBlockGCOp(9f3cbedfa1da42698615a544d9dc3276) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4}
I20240502 14:16:29.921500 26604 maintenance_manager.cc:419] P ba187e38a2754d60a12de9ac4856430a: Scheduling UndoDeltaBlockGCOp(d7f02f897ce54fefb03832da3c2d95e0): 4840 bytes on disk
I20240502 14:16:29.923000 26495 maintenance_manager.cc:643] P ba187e38a2754d60a12de9ac4856430a: UndoDeltaBlockGCOp(d7f02f897ce54fefb03832da3c2d95e0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
I20240502 14:16:30.089135 29574 consensus_queue.cc:237] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:30.091724 29440 raft_consensus.cc:1261] T 23f576e1bede478496122a9d1a2e7515 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:30.091886 26427 raft_consensus.cc:1261] T 23f576e1bede478496122a9d1a2e7515 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:30.092507 8986 consensus_queue.cc:1035] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:30.093027 8986 consensus_queue.cc:1035] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:30.095999 8986 raft_consensus.cc:2932] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:30.107123 29440 raft_consensus.cc:2932] T 23f576e1bede478496122a9d1a2e7515 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:30.115011 26285 catalog_manager.cc:5484] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:30.126754 26427 raft_consensus.cc:2932] T 23f576e1bede478496122a9d1a2e7515 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:30.138402 29287 tablet_service.cc:1514] Processing DeleteTablet for tablet 23f576e1bede478496122a9d1a2e7515 with delete_type TABLET_DATA_TOMBSTONED (TS c1100e16b3b9495ca61d4f95e89b6a89 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:59160
I20240502 14:16:30.139672 9387 tablet_replica.cc:330] T 23f576e1bede478496122a9d1a2e7515 P c1100e16b3b9495ca61d4f95e89b6a89: stopping tablet replica
I20240502 14:16:30.139993 9387 raft_consensus.cc:2229] T 23f576e1bede478496122a9d1a2e7515 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:30.140425 9387 raft_consensus.cc:2258] T 23f576e1bede478496122a9d1a2e7515 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:30.141539 9387 ts_tablet_manager.cc:1875] T 23f576e1bede478496122a9d1a2e7515 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:30.151226 9387 ts_tablet_manager.cc:1888] T 23f576e1bede478496122a9d1a2e7515 P c1100e16b3b9495ca61d4f95e89b6a89: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:16:30.151502 9387 log.cc:1196] T 23f576e1bede478496122a9d1a2e7515 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal/wals/23f576e1bede478496122a9d1a2e7515
I20240502 14:16:30.152403 26271 catalog_manager.cc:4830] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): tablet 23f576e1bede478496122a9d1a2e7515 (table rebalancer_extra_table_38 [id=7a3bd56fd0ef4a5f89e69991dd8d785a]) successfully deleted
I20240502 14:16:30.534282 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:30.553504 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:16:30.572769 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:30.573509 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:30.581023 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:30.592943 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:30.633201 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:30.636771 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:30.829155 26427 consensus_queue.cc:237] T a503853cfec1454f9a5862d1a8e5ead5 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:16:30.832113 26816 raft_consensus.cc:1261] T a503853cfec1454f9a5862d1a8e5ead5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:30.832296 29176 raft_consensus.cc:1261] T a503853cfec1454f9a5862d1a8e5ead5 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:30.832942 9440 consensus_queue.cc:1035] T a503853cfec1454f9a5862d1a8e5ead5 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:30.837687 9440 consensus_queue.cc:1035] T a503853cfec1454f9a5862d1a8e5ead5 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:30.838246 9440 raft_consensus.cc:2932] T a503853cfec1454f9a5862d1a8e5ead5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:30.841537 29176 raft_consensus.cc:2932] T a503853cfec1454f9a5862d1a8e5ead5 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:30.847812 26277 catalog_manager.cc:5484] T a503853cfec1454f9a5862d1a8e5ead5 P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:30.880837 29421 tablet_service.cc:1514] Processing DeleteTablet for tablet a503853cfec1454f9a5862d1a8e5ead5 with delete_type TABLET_DATA_TOMBSTONED (TS 68a3c2b17ed94e06aba66ed4178132ab not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:40652
I20240502 14:16:30.882136 26816 raft_consensus.cc:2932] T a503853cfec1454f9a5862d1a8e5ead5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:30.893268 9469 tablet_replica.cc:330] T a503853cfec1454f9a5862d1a8e5ead5 P 68a3c2b17ed94e06aba66ed4178132ab: stopping tablet replica
I20240502 14:16:30.893486 9469 raft_consensus.cc:2229] T a503853cfec1454f9a5862d1a8e5ead5 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:30.893676 9469 raft_consensus.cc:2258] T a503853cfec1454f9a5862d1a8e5ead5 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:30.894251 9469 ts_tablet_manager.cc:1875] T a503853cfec1454f9a5862d1a8e5ead5 P 68a3c2b17ed94e06aba66ed4178132ab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:30.902122 9469 ts_tablet_manager.cc:1888] T a503853cfec1454f9a5862d1a8e5ead5 P 68a3c2b17ed94e06aba66ed4178132ab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:16:30.902992 9469 log.cc:1196] T a503853cfec1454f9a5862d1a8e5ead5 P 68a3c2b17ed94e06aba66ed4178132ab: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal/wals/a503853cfec1454f9a5862d1a8e5ead5
I20240502 14:16:30.907608 26272 catalog_manager.cc:4830] TS 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): tablet a503853cfec1454f9a5862d1a8e5ead5 (table rebalancer_extra_table_36 [id=123f30b9ad634d07ada549c46031d008]) successfully deleted
I20240502 14:16:31.318993 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:31.345430 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:31.360785 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:16:31.391147 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:31.461875 26604 maintenance_manager.cc:419] P ba187e38a2754d60a12de9ac4856430a: Scheduling FlushMRSOp(825ae3e90e214a6f98394765edf6ff2d): perf score=0.033338
I20240502 14:16:31.481480 26473 maintenance_manager.cc:419] P 89af7f423cf44717a08d6b142f0ed49d: Scheduling FlushMRSOp(825ae3e90e214a6f98394765edf6ff2d): perf score=0.033348
I20240502 14:16:31.484169 26735 maintenance_manager.cc:419] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Scheduling FlushMRSOp(310aa6ad3604487589dbe96afa25bb05): perf score=0.033351
I20240502 14:16:31.530252 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:31.562605 26865 maintenance_manager.cc:419] P e4cfb8fc5a474177af6848a6efa13a27: Scheduling FlushMRSOp(2108a04310804945b368bcb5f88170e2): perf score=0.033368
I20240502 14:16:31.663197 26626 maintenance_manager.cc:643] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: FlushMRSOp(310aa6ad3604487589dbe96afa25bb05) complete. Timing: real 0.179s user 0.054s sys 0.063s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":9939,"dirs.run_cpu_time_us":246,"dirs.run_wall_time_us":6637,"drs_written":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1205,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":109780,"threads_started":2,"wal-append.queue_time_us":99854}
I20240502 14:16:31.667230 26735 maintenance_manager.cc:419] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Scheduling FlushMRSOp(2108a04310804945b368bcb5f88170e2): perf score=0.033398
I20240502 14:16:31.684842 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:31.700644 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:31.713166 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:31.835318 26626 maintenance_manager.cc:643] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: FlushMRSOp(2108a04310804945b368bcb5f88170e2) complete. Timing: real 0.159s user 0.056s sys 0.021s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":47,"dirs.run_cpu_time_us":218,"dirs.run_wall_time_us":1442,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":902,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":27520,"thread_start_us":6410,"threads_started":1,"wal-append.queue_time_us":6517}
I20240502 14:16:31.838609 26735 maintenance_manager.cc:419] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Scheduling UndoDeltaBlockGCOp(2108a04310804945b368bcb5f88170e2): 4925 bytes on disk
I20240502 14:16:31.839390 26626 maintenance_manager.cc:643] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: UndoDeltaBlockGCOp(2108a04310804945b368bcb5f88170e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4}
I20240502 14:16:31.843166 26735 maintenance_manager.cc:419] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Scheduling UndoDeltaBlockGCOp(310aa6ad3604487589dbe96afa25bb05): 4586 bytes on disk
I20240502 14:16:31.843910 26626 maintenance_manager.cc:643] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: UndoDeltaBlockGCOp(310aa6ad3604487589dbe96afa25bb05) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":187,"lbm_reads_lt_1ms":4}
I20240502 14:16:31.900851 26364 maintenance_manager.cc:643] P 89af7f423cf44717a08d6b142f0ed49d: FlushMRSOp(825ae3e90e214a6f98394765edf6ff2d) complete. Timing: real 0.419s user 0.104s sys 0.045s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":8729,"dirs.run_cpu_time_us":271,"dirs.run_wall_time_us":11051,"drs_written":1,"lbm_read_time_us":178,"lbm_reads_lt_1ms":4,"lbm_write_time_us":934,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":77328,"threads_started":2,"wal-append.queue_time_us":68516}
I20240502 14:16:31.905364 26473 maintenance_manager.cc:419] P 89af7f423cf44717a08d6b142f0ed49d: Scheduling UndoDeltaBlockGCOp(825ae3e90e214a6f98394765edf6ff2d): 4840 bytes on disk
I20240502 14:16:31.907241 26364 maintenance_manager.cc:643] P 89af7f423cf44717a08d6b142f0ed49d: UndoDeltaBlockGCOp(825ae3e90e214a6f98394765edf6ff2d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":118,"lbm_reads_lt_1ms":4}
I20240502 14:16:31.949734 26495 maintenance_manager.cc:643] P ba187e38a2754d60a12de9ac4856430a: FlushMRSOp(825ae3e90e214a6f98394765edf6ff2d) complete. Timing: real 0.487s user 0.033s sys 0.037s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":681,"dirs.run_cpu_time_us":308,"dirs.run_wall_time_us":2144,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":874,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":39186,"threads_started":2,"wal-append.queue_time_us":14161}
I20240502 14:16:31.952654 26604 maintenance_manager.cc:419] P ba187e38a2754d60a12de9ac4856430a: Scheduling UndoDeltaBlockGCOp(825ae3e90e214a6f98394765edf6ff2d): 4840 bytes on disk
I20240502 14:16:31.953405 26495 maintenance_manager.cc:643] P ba187e38a2754d60a12de9ac4856430a: UndoDeltaBlockGCOp(825ae3e90e214a6f98394765edf6ff2d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":131,"lbm_reads_lt_1ms":4}
I20240502 14:16:31.955971 26604 maintenance_manager.cc:419] P ba187e38a2754d60a12de9ac4856430a: Scheduling FlushMRSOp(310aa6ad3604487589dbe96afa25bb05): perf score=0.033473
I20240502 14:16:31.999279 26756 maintenance_manager.cc:643] P e4cfb8fc5a474177af6848a6efa13a27: FlushMRSOp(2108a04310804945b368bcb5f88170e2) complete. Timing: real 0.436s user 0.071s sys 0.009s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":9003,"dirs.run_cpu_time_us":246,"dirs.run_wall_time_us":1447,"drs_written":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1122,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":9182,"threads_started":2,"wal-append.queue_time_us":337}
I20240502 14:16:32.004248 26865 maintenance_manager.cc:419] P e4cfb8fc5a474177af6848a6efa13a27: Scheduling UndoDeltaBlockGCOp(2108a04310804945b368bcb5f88170e2): 4925 bytes on disk
I20240502 14:16:32.005000 26756 maintenance_manager.cc:643] P e4cfb8fc5a474177af6848a6efa13a27: UndoDeltaBlockGCOp(2108a04310804945b368bcb5f88170e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4}
I20240502 14:16:32.060101 26495 maintenance_manager.cc:643] P ba187e38a2754d60a12de9ac4856430a: FlushMRSOp(310aa6ad3604487589dbe96afa25bb05) complete. Timing: real 0.104s user 0.067s sys 0.014s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":75,"dirs.run_cpu_time_us":225,"dirs.run_wall_time_us":1540,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1137,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":12634,"threads_started":1,"wal-append.queue_time_us":12735}
I20240502 14:16:32.062855 26604 maintenance_manager.cc:419] P ba187e38a2754d60a12de9ac4856430a: Scheduling UndoDeltaBlockGCOp(310aa6ad3604487589dbe96afa25bb05): 4586 bytes on disk
I20240502 14:16:32.063499 26495 maintenance_manager.cc:643] P ba187e38a2754d60a12de9ac4856430a: UndoDeltaBlockGCOp(310aa6ad3604487589dbe96afa25bb05) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4}
I20240502 14:16:32.068480 26815 consensus_queue.cc:237] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:16:32.071192 26557 raft_consensus.cc:1261] T 4df2bd4e182c4fe991da385d92d3f76f P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:32.071280 29443 raft_consensus.cc:1261] T 4df2bd4e182c4fe991da385d92d3f76f P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:32.071902 9547 consensus_queue.cc:1035] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:32.072240 9519 consensus_queue.cc:1035] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:32.105609 9212 raft_consensus.cc:2932] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:32.106420 26557 raft_consensus.cc:2932] T 4df2bd4e182c4fe991da385d92d3f76f P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:32.106405 29443 raft_consensus.cc:2932] T 4df2bd4e182c4fe991da385d92d3f76f P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:32.113165 26282 catalog_manager.cc:5484] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: config changed from index 4 to 5, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:32.139425 26669 tablet_service.cc:1514] Processing DeleteTablet for tablet 4df2bd4e182c4fe991da385d92d3f76f with delete_type TABLET_DATA_TOMBSTONED (TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34150
I20240502 14:16:32.141839 9586 tablet_replica.cc:330] T 4df2bd4e182c4fe991da385d92d3f76f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: stopping tablet replica
I20240502 14:16:32.142082 9586 raft_consensus.cc:2229] T 4df2bd4e182c4fe991da385d92d3f76f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:32.142325 9586 raft_consensus.cc:2258] T 4df2bd4e182c4fe991da385d92d3f76f P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:32.143157 9586 ts_tablet_manager.cc:1875] T 4df2bd4e182c4fe991da385d92d3f76f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:32.171237 9586 ts_tablet_manager.cc:1888] T 4df2bd4e182c4fe991da385d92d3f76f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:16:32.171618 9586 log.cc:1196] T 4df2bd4e182c4fe991da385d92d3f76f P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/wals/4df2bd4e182c4fe991da385d92d3f76f
I20240502 14:16:32.172509 26272 catalog_manager.cc:4830] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): tablet 4df2bd4e182c4fe991da385d92d3f76f (table rebalance_test_table_11 [id=c7861c8f192f462e8ed2198ec9c0c5ea]) successfully deleted
I20240502 14:16:32.475781 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:32.513415 29155 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:32.525209 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:32.588929 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:32.635443 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:32.676921 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:16:32.700553 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:32.769590 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:32.880257 26556 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5e66bc4ba1f84b50884d6492edb1e0d2"
dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36862
I20240502 14:16:32.880522 26556 raft_consensus.cc:595] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Received request to transfer leadership
I20240502 14:16:33.016299 9635 raft_consensus.cc:979] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a: : Instructing follower 1cb67158133e4648a6eceba32d4dbba3 to start an election
I20240502 14:16:33.016492 9635 raft_consensus.cc:1067] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Signalling peer 1cb67158133e4648a6eceba32d4dbba3 to start an election
I20240502 14:16:33.018992 29176 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5e66bc4ba1f84b50884d6492edb1e0d2"
dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
from {username='slave'} at 127.19.94.194:50529
I20240502 14:16:33.019264 29176 raft_consensus.cc:482] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:16:33.019376 29176 raft_consensus.cc:3037] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:33.022302 29176 raft_consensus.cc:504] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:16:33.029685 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e66bc4ba1f84b50884d6492edb1e0d2" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
I20240502 14:16:33.029953 29443 raft_consensus.cc:3037] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:33.031610 26556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e66bc4ba1f84b50884d6492edb1e0d2" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:16:33.031908 26556 raft_consensus.cc:3032] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:16:33.031988 26556 raft_consensus.cc:729] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Running, Role: LEADER
I20240502 14:16:33.032348 26556 consensus_queue.cc:260] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:16:33.032955 26556 raft_consensus.cc:3037] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:33.033289 29443 raft_consensus.cc:2454] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 68a3c2b17ed94e06aba66ed4178132ab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 2.
I20240502 14:16:33.035190 29176 leader_election.cc:288] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 2 election: Requested vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:16:33.035620 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e66bc4ba1f84b50884d6492edb1e0d2" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
I20240502 14:16:33.035889 29309 raft_consensus.cc:3037] T 5e66bc4ba1f84b50884d6492edb1e0d2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:33.042987 29309 raft_consensus.cc:2454] T 5e66bc4ba1f84b50884d6492edb1e0d2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 2.
I20240502 14:16:33.043951 29111 leader_election.cc:302] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89; no voters:
I20240502 14:16:33.044620 9250 raft_consensus.cc:2781] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:16:33.044761 9250 raft_consensus.cc:686] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 [term 2 LEADER]: Becoming Leader. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Running, Role: LEADER
I20240502 14:16:33.045033 9250 consensus_queue.cc:237] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:16:33.044987 26556 raft_consensus.cc:2454] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 2.
I20240502 14:16:33.050957 26284 catalog_manager.cc:5484] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: term changed from 1 to 2, leader changed from ba187e38a2754d60a12de9ac4856430a (127.19.94.194) to 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197). New cstate: current_term: 2 leader_uuid: "1cb67158133e4648a6eceba32d4dbba3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:33.321441 29155 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:33.356976 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:33.361477 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:33.412791 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:33.473569 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:33.473775 26538 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:16:33.482085 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:16:33.498593 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:33.593895 26557 raft_consensus.cc:1261] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a [term 2 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:16:33.594451 9683 consensus_queue.cc:1035] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:33.631083 29440 raft_consensus.cc:1261] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 68a3c2b17ed94e06aba66ed4178132ab [term 2 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:16:33.631793 9683 consensus_queue.cc:1035] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:33.639231 29309 raft_consensus.cc:1261] T 5e66bc4ba1f84b50884d6492edb1e0d2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 2 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:16:33.641093 9250 consensus_queue.cc:1035] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:34.218428 29155 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:34.254081 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240502 14:16:34.312199 29112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 54621.2us
I20240502 14:16:34.330029 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:34.362932 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:34.406690 26538 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:16:34.425163 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:34.440805 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:34.464125 26406 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:16:34.600838 26425 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b0316714f34d44a5b537cdf20af42ce9"
dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37262
I20240502 14:16:34.601143 26425 raft_consensus.cc:595] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Received request to transfer leadership
I20240502 14:16:34.737496 9440 raft_consensus.cc:979] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d: : Instructing follower ba187e38a2754d60a12de9ac4856430a to start an election
I20240502 14:16:34.737682 9440 raft_consensus.cc:1067] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Signalling peer ba187e38a2754d60a12de9ac4856430a to start an election
I20240502 14:16:34.738622 26556 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b0316714f34d44a5b537cdf20af42ce9"
dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
from {username='slave'} at 127.19.94.193:46185
I20240502 14:16:34.738881 26556 raft_consensus.cc:482] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:16:34.738996 26556 raft_consensus.cc:3037] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:34.742130 26556 raft_consensus.cc:504] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:16:34.748180 26816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0316714f34d44a5b537cdf20af42ce9" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:16:34.748471 26816 raft_consensus.cc:3037] T b0316714f34d44a5b537cdf20af42ce9 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:34.751739 26816 raft_consensus.cc:2454] T b0316714f34d44a5b537cdf20af42ce9 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 2.
I20240502 14:16:34.755141 26425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0316714f34d44a5b537cdf20af42ce9" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:16:34.755383 26425 raft_consensus.cc:3032] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:16:34.755463 26425 raft_consensus.cc:729] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Running, Role: LEADER
I20240502 14:16:34.755818 26425 consensus_queue.cc:260] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:16:34.756328 26425 raft_consensus.cc:3037] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:34.766877 26425 raft_consensus.cc:2454] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 2.
I20240502 14:16:34.767453 26493 leader_election.cc:302] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, ba187e38a2754d60a12de9ac4856430a, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:16:34.767827 9251 raft_consensus.cc:2781] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:16:34.767946 9251 raft_consensus.cc:686] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a [term 2 LEADER]: Becoming Leader. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Running, Role: LEADER
I20240502 14:16:34.768239 9251 consensus_queue.cc:237] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:16:34.769974 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0316714f34d44a5b537cdf20af42ce9" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "c1100e16b3b9495ca61d4f95e89b6a89"
I20240502 14:16:34.770282 29309 raft_consensus.cc:3037] T b0316714f34d44a5b537cdf20af42ce9 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:34.784282 26284 catalog_manager.cc:5484] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a reported cstate change: term changed from 1 to 2, leader changed from 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) to ba187e38a2754d60a12de9ac4856430a (127.19.94.194). New cstate: current_term: 2 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:34.786020 26556 leader_election.cc:288] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 election: Requested vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:16:34.784472 29309 raft_consensus.cc:2454] T b0316714f34d44a5b537cdf20af42ce9 P c1100e16b3b9495ca61d4f95e89b6a89 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 2.
I20240502 14:16:34.813936 26865 maintenance_manager.cc:419] P e4cfb8fc5a474177af6848a6efa13a27: Scheduling FlushMRSOp(4df2bd4e182c4fe991da385d92d3f76f): perf score=0.033334
I20240502 14:16:34.853478 26604 maintenance_manager.cc:419] P ba187e38a2754d60a12de9ac4856430a: Scheduling FlushMRSOp(d1aa4f4be5894ec5a64275f1172e1f55): perf score=0.033352
I20240502 14:16:34.880167 26735 maintenance_manager.cc:419] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Scheduling FlushMRSOp(0290ebbcf609427cae8fce6637019336): perf score=0.033362
I20240502 14:16:34.958978 26756 maintenance_manager.cc:643] P e4cfb8fc5a474177af6848a6efa13a27: FlushMRSOp(4df2bd4e182c4fe991da385d92d3f76f) complete. Timing: real 0.145s user 0.074s sys 0.019s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":5554,"dirs.run_cpu_time_us":347,"dirs.run_wall_time_us":4267,"drs_written":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1243,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":73561,"threads_started":2,"wal-append.queue_time_us":66990}
I20240502 14:16:34.962190 26865 maintenance_manager.cc:419] P e4cfb8fc5a474177af6848a6efa13a27: Scheduling UndoDeltaBlockGCOp(4df2bd4e182c4fe991da385d92d3f76f): 4586 bytes on disk
I20240502 14:16:34.964993 26473 maintenance_manager.cc:419] P 89af7f423cf44717a08d6b142f0ed49d: Scheduling FlushMRSOp(d1aa4f4be5894ec5a64275f1172e1f55): perf score=0.033382
I20240502 14:16:34.969431 26756 maintenance_manager.cc:643] P e4cfb8fc5a474177af6848a6efa13a27: UndoDeltaBlockGCOp(4df2bd4e182c4fe991da385d92d3f76f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":934,"lbm_reads_lt_1ms":4}
I20240502 14:16:35.098392 26495 maintenance_manager.cc:643] P ba187e38a2754d60a12de9ac4856430a: FlushMRSOp(d1aa4f4be5894ec5a64275f1172e1f55) complete. Timing: real 0.243s user 0.072s sys 0.024s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":19181,"dirs.run_cpu_time_us":252,"dirs.run_wall_time_us":1849,"drs_written":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4,"lbm_write_time_us":10690,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":35213,"threads_started":2,"wal-append.queue_time_us":15711}
I20240502 14:16:35.102138 26604 maintenance_manager.cc:419] P ba187e38a2754d60a12de9ac4856430a: Scheduling UndoDeltaBlockGCOp(d1aa4f4be5894ec5a64275f1172e1f55): 4925 bytes on disk
I20240502 14:16:35.110453 26495 maintenance_manager.cc:643] P ba187e38a2754d60a12de9ac4856430a: UndoDeltaBlockGCOp(d1aa4f4be5894ec5a64275f1172e1f55) complete. Timing: real 0.003s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":2863,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3}
I20240502 14:16:35.114048 26604 maintenance_manager.cc:419] P ba187e38a2754d60a12de9ac4856430a: Scheduling FlushMRSOp(4df2bd4e182c4fe991da385d92d3f76f): perf score=0.033418
I20240502 14:16:35.149701 29155 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:35.166947 26425 raft_consensus.cc:1261] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:16:35.168131 9251 consensus_queue.cc:1035] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:35.184700 29307 raft_consensus.cc:1261] T b0316714f34d44a5b537cdf20af42ce9 P c1100e16b3b9495ca61d4f95e89b6a89 [term 2 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:16:35.193625 9251 consensus_queue.cc:1035] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:35.253175 26816 raft_consensus.cc:1261] T b0316714f34d44a5b537cdf20af42ce9 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:16:35.254411 9251 consensus_queue.cc:1035] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:35.333987 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:35.428526 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:35.430931 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:35.504520 26626 maintenance_manager.cc:643] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: FlushMRSOp(0290ebbcf609427cae8fce6637019336) complete. Timing: real 0.624s user 0.071s sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":15473,"dirs.run_cpu_time_us":266,"dirs.run_wall_time_us":14002,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4970,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":77606,"threads_started":2,"wal-append.queue_time_us":60355}
I20240502 14:16:35.523039 26735 maintenance_manager.cc:419] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Scheduling UndoDeltaBlockGCOp(0290ebbcf609427cae8fce6637019336): 4840 bytes on disk
I20240502 14:16:35.523819 26626 maintenance_manager.cc:643] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: UndoDeltaBlockGCOp(0290ebbcf609427cae8fce6637019336) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":121,"lbm_reads_lt_1ms":4}
I20240502 14:16:35.553153 26406 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:35.553982 26495 maintenance_manager.cc:643] P ba187e38a2754d60a12de9ac4856430a: FlushMRSOp(4df2bd4e182c4fe991da385d92d3f76f) complete. Timing: real 0.438s user 0.047s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":88,"dirs.run_cpu_time_us":196,"dirs.run_wall_time_us":1407,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":7360,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":21,"peak_mem_usage":0,"rows_written":635,"thread_start_us":129756,"threads_started":1,"wal-append.queue_time_us":129872}
I20240502 14:16:35.560704 26604 maintenance_manager.cc:419] P ba187e38a2754d60a12de9ac4856430a: Scheduling UndoDeltaBlockGCOp(4df2bd4e182c4fe991da385d92d3f76f): 4586 bytes on disk
I20240502 14:16:35.561671 26495 maintenance_manager.cc:643] P ba187e38a2754d60a12de9ac4856430a: UndoDeltaBlockGCOp(4df2bd4e182c4fe991da385d92d3f76f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":126,"lbm_reads_lt_1ms":4}
I20240502 14:16:35.567852 26364 maintenance_manager.cc:643] P 89af7f423cf44717a08d6b142f0ed49d: FlushMRSOp(d1aa4f4be5894ec5a64275f1172e1f55) complete. Timing: real 0.603s user 0.122s sys 0.059s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":6916,"dirs.run_cpu_time_us":275,"dirs.run_wall_time_us":12193,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1061,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":136381,"threads_started":2,"wal-append.queue_time_us":128358}
I20240502 14:16:35.570802 26473 maintenance_manager.cc:419] P 89af7f423cf44717a08d6b142f0ed49d: Scheduling FlushMRSOp(0290ebbcf609427cae8fce6637019336): perf score=0.033547
I20240502 14:16:35.693089 26364 maintenance_manager.cc:643] P 89af7f423cf44717a08d6b142f0ed49d: FlushMRSOp(0290ebbcf609427cae8fce6637019336) complete. Timing: real 0.119s user 0.054s sys 0.016s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":64,"dirs.run_cpu_time_us":257,"dirs.run_wall_time_us":1526,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1043,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"spinlock_wait_cycles":3968,"thread_start_us":26292,"threads_started":1,"wal-append.queue_time_us":26814}
I20240502 14:16:35.700592 26473 maintenance_manager.cc:419] P 89af7f423cf44717a08d6b142f0ed49d: Scheduling UndoDeltaBlockGCOp(d1aa4f4be5894ec5a64275f1172e1f55): 4925 bytes on disk
I20240502 14:16:35.701318 26364 maintenance_manager.cc:643] P 89af7f423cf44717a08d6b142f0ed49d: UndoDeltaBlockGCOp(d1aa4f4be5894ec5a64275f1172e1f55) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4}
I20240502 14:16:35.709333 26473 maintenance_manager.cc:419] P 89af7f423cf44717a08d6b142f0ed49d: Scheduling UndoDeltaBlockGCOp(0290ebbcf609427cae8fce6637019336): 4840 bytes on disk
I20240502 14:16:35.710162 26364 maintenance_manager.cc:643] P 89af7f423cf44717a08d6b142f0ed49d: UndoDeltaBlockGCOp(0290ebbcf609427cae8fce6637019336) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
I20240502 14:16:35.716290 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:35.731289 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:35.749228 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:35.945732 26556 consensus_queue.cc:237] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:16:35.947888 29307 raft_consensus.cc:1261] T b0316714f34d44a5b537cdf20af42ce9 P c1100e16b3b9495ca61d4f95e89b6a89 [term 2 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:16:35.948510 9635 consensus_queue.cc:1035] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:35.949409 26815 raft_consensus.cc:1261] T b0316714f34d44a5b537cdf20af42ce9 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:16:35.949941 9635 consensus_queue.cc:1035] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:35.950784 9251 raft_consensus.cc:2932] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:35.951414 29307 raft_consensus.cc:2932] T b0316714f34d44a5b537cdf20af42ce9 P c1100e16b3b9495ca61d4f95e89b6a89 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:35.953598 26816 raft_consensus.cc:2932] T b0316714f34d44a5b537cdf20af42ce9 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:35.959467 26280 catalog_manager.cc:5484] T b0316714f34d44a5b537cdf20af42ce9 P c1100e16b3b9495ca61d4f95e89b6a89 reported cstate change: config changed from index 2 to 4, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) evicted. New cstate: current_term: 2 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:35.989719 26406 tablet_service.cc:1514] Processing DeleteTablet for tablet b0316714f34d44a5b537cdf20af42ce9 with delete_type TABLET_DATA_TOMBSTONED (TS 89af7f423cf44717a08d6b142f0ed49d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37422
I20240502 14:16:35.991781 9935 tablet_replica.cc:330] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d: stopping tablet replica
I20240502 14:16:35.992115 9935 raft_consensus.cc:2229] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:35.992440 9935 raft_consensus.cc:2258] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:35.994179 9935 ts_tablet_manager.cc:1875] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:36.003357 9935 ts_tablet_manager.cc:1888] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240502 14:16:36.003571 9935 log.cc:1196] T b0316714f34d44a5b537cdf20af42ce9 P 89af7f423cf44717a08d6b142f0ed49d: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/b0316714f34d44a5b537cdf20af42ce9
I20240502 14:16:36.005972 26271 catalog_manager.cc:4830] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): tablet b0316714f34d44a5b537cdf20af42ce9 (table rebalancer_extra_table_44 [id=d20a7e521c7f4f93a17f7a49aa1c5d28]) successfully deleted
I20240502 14:16:36.420128 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:36.421393 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:36.421880 29155 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:36.473062 26406 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:36.499889 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:36.509889 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:36.511209 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:36.531060 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:36.757078 26686 consensus_queue.cc:237] T e665b144dfe541dfaf492281d2f72100 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:16:36.759280 26557 raft_consensus.cc:1261] T e665b144dfe541dfaf492281d2f72100 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:36.759372 26816 raft_consensus.cc:1261] T e665b144dfe541dfaf492281d2f72100 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:36.759877 9621 consensus_queue.cc:1035] T e665b144dfe541dfaf492281d2f72100 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:36.760157 9465 consensus_queue.cc:1035] T e665b144dfe541dfaf492281d2f72100 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:36.764271 9465 raft_consensus.cc:2932] T e665b144dfe541dfaf492281d2f72100 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:36.768417 26557 raft_consensus.cc:2932] T e665b144dfe541dfaf492281d2f72100 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:36.771471 26816 raft_consensus.cc:2932] T e665b144dfe541dfaf492281d2f72100 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:36.771701 26284 catalog_manager.cc:5484] T e665b144dfe541dfaf492281d2f72100 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:36.792848 29421 tablet_service.cc:1514] Processing DeleteTablet for tablet e665b144dfe541dfaf492281d2f72100 with delete_type TABLET_DATA_TOMBSTONED (TS 68a3c2b17ed94e06aba66ed4178132ab not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:40652
I20240502 14:16:36.793527 10031 tablet_replica.cc:330] T e665b144dfe541dfaf492281d2f72100 P 68a3c2b17ed94e06aba66ed4178132ab: stopping tablet replica
I20240502 14:16:36.793718 10031 raft_consensus.cc:2229] T e665b144dfe541dfaf492281d2f72100 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:36.793916 10031 raft_consensus.cc:2258] T e665b144dfe541dfaf492281d2f72100 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:36.794588 10031 ts_tablet_manager.cc:1875] T e665b144dfe541dfaf492281d2f72100 P 68a3c2b17ed94e06aba66ed4178132ab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:36.800571 10031 ts_tablet_manager.cc:1888] T e665b144dfe541dfaf492281d2f72100 P 68a3c2b17ed94e06aba66ed4178132ab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:16:36.800705 10031 log.cc:1196] T e665b144dfe541dfaf492281d2f72100 P 68a3c2b17ed94e06aba66ed4178132ab: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal/wals/e665b144dfe541dfaf492281d2f72100
I20240502 14:16:36.801537 26272 catalog_manager.cc:4830] TS 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): tablet e665b144dfe541dfaf492281d2f72100 (table rebalancer_extra_table_46 [id=cf2fa69bdc0c4ba4af966938f6e97bd7]) successfully deleted
I20240502 14:16:37.112286 29155 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:16:37.207811 26755 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 73796.2us
I20240502 14:16:37.209365 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:37.210115 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:37.244609 26473 maintenance_manager.cc:419] P 89af7f423cf44717a08d6b142f0ed49d: Scheduling FlushMRSOp(1fb1ef75829641a89c4cac539f431d80): perf score=0.033349
I20240502 14:16:37.252064 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:37.251969 26865 maintenance_manager.cc:419] P e4cfb8fc5a474177af6848a6efa13a27: Scheduling FlushMRSOp(c41522c60f3942aa9f28f514dcad4e49): perf score=0.033357
I20240502 14:16:37.271020 26406 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:37.288173 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:37.309422 26735 maintenance_manager.cc:419] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Scheduling FlushMRSOp(c41522c60f3942aa9f28f514dcad4e49): perf score=0.033375
I20240502 14:16:37.348043 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:37.355083 26604 maintenance_manager.cc:419] P ba187e38a2754d60a12de9ac4856430a: Scheduling FlushMRSOp(6814613d36a74353a99673cf16aa9821): perf score=0.033386
I20240502 14:16:37.415599 26626 maintenance_manager.cc:643] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: FlushMRSOp(c41522c60f3942aa9f28f514dcad4e49) complete. Timing: real 0.106s user 0.039s sys 0.015s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":8960,"dirs.run_cpu_time_us":243,"dirs.run_wall_time_us":7522,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":839,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":25882,"threads_started":2,"wal-append.queue_time_us":17200}
I20240502 14:16:37.425101 26735 maintenance_manager.cc:419] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Scheduling FlushMRSOp(1fb1ef75829641a89c4cac539f431d80): perf score=0.033402
I20240502 14:16:37.451638 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:37.586311 26756 maintenance_manager.cc:643] P e4cfb8fc5a474177af6848a6efa13a27: FlushMRSOp(c41522c60f3942aa9f28f514dcad4e49) complete. Timing: real 0.328s user 0.049s sys 0.029s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":15309,"dirs.run_cpu_time_us":231,"dirs.run_wall_time_us":2770,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":981,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":118381,"threads_started":2,"wal-append.queue_time_us":78104}
I20240502 14:16:37.629999 26865 maintenance_manager.cc:419] P e4cfb8fc5a474177af6848a6efa13a27: Scheduling UndoDeltaBlockGCOp(c41522c60f3942aa9f28f514dcad4e49): 4925 bytes on disk
I20240502 14:16:37.673401 26364 maintenance_manager.cc:643] P 89af7f423cf44717a08d6b142f0ed49d: FlushMRSOp(1fb1ef75829641a89c4cac539f431d80) complete. Timing: real 0.429s user 0.041s sys 0.009s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":3764,"dirs.run_cpu_time_us":240,"dirs.run_wall_time_us":1530,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":801,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":16609,"threads_started":2,"wal-append.queue_time_us":3680}
I20240502 14:16:37.685292 26473 maintenance_manager.cc:419] P 89af7f423cf44717a08d6b142f0ed49d: Scheduling UndoDeltaBlockGCOp(1fb1ef75829641a89c4cac539f431d80): 4586 bytes on disk
I20240502 14:16:37.686115 26364 maintenance_manager.cc:643] P 89af7f423cf44717a08d6b142f0ed49d: UndoDeltaBlockGCOp(1fb1ef75829641a89c4cac539f431d80) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":153,"lbm_reads_lt_1ms":4}
I20240502 14:16:37.688689 26473 maintenance_manager.cc:419] P 89af7f423cf44717a08d6b142f0ed49d: Scheduling FlushMRSOp(6814613d36a74353a99673cf16aa9821): perf score=0.033471
I20240502 14:16:37.725867 26756 maintenance_manager.cc:643] P e4cfb8fc5a474177af6848a6efa13a27: UndoDeltaBlockGCOp(c41522c60f3942aa9f28f514dcad4e49) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":634,"lbm_reads_lt_1ms":4}
I20240502 14:16:37.770161 26495 maintenance_manager.cc:643] P ba187e38a2754d60a12de9ac4856430a: FlushMRSOp(6814613d36a74353a99673cf16aa9821) complete. Timing: real 0.415s user 0.099s sys 0.073s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":18660,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":2109,"drs_written":1,"lbm_read_time_us":938,"lbm_reads_lt_1ms":4,"lbm_write_time_us":970,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":164174,"threads_started":2,"wal-append.queue_time_us":145752}
I20240502 14:16:37.782850 26604 maintenance_manager.cc:419] P ba187e38a2754d60a12de9ac4856430a: Scheduling UndoDeltaBlockGCOp(6814613d36a74353a99673cf16aa9821): 4840 bytes on disk
I20240502 14:16:37.783538 26495 maintenance_manager.cc:643] P ba187e38a2754d60a12de9ac4856430a: UndoDeltaBlockGCOp(6814613d36a74353a99673cf16aa9821) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4}
I20240502 14:16:37.851414 26364 maintenance_manager.cc:643] P 89af7f423cf44717a08d6b142f0ed49d: FlushMRSOp(6814613d36a74353a99673cf16aa9821) complete. Timing: real 0.162s user 0.051s sys 0.042s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":48,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":1442,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":914,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":43368,"threads_started":1,"wal-append.queue_time_us":43248}
I20240502 14:16:37.854522 26473 maintenance_manager.cc:419] P 89af7f423cf44717a08d6b142f0ed49d: Scheduling UndoDeltaBlockGCOp(6814613d36a74353a99673cf16aa9821): 4840 bytes on disk
I20240502 14:16:37.855216 26364 maintenance_manager.cc:643] P 89af7f423cf44717a08d6b142f0ed49d: UndoDeltaBlockGCOp(6814613d36a74353a99673cf16aa9821) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
W20240502 14:16:37.857784 26626 tablet_replica.cc:1402] Time spent flushing the Log queue took a long time: real 0.262s user 0.085s sys 0.101s
I20240502 14:16:37.862322 26626 maintenance_manager.cc:643] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: FlushMRSOp(1fb1ef75829641a89c4cac539f431d80) complete. Timing: real 0.434s user 0.140s sys 0.115s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":55,"dirs.run_cpu_time_us":206,"dirs.run_wall_time_us":1570,"drs_written":1,"lbm_read_time_us":9825,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3,"lbm_write_time_us":979,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":261414,"threads_started":1,"wal-append.queue_time_us":257153}
I20240502 14:16:37.864867 26735 maintenance_manager.cc:419] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Scheduling UndoDeltaBlockGCOp(c41522c60f3942aa9f28f514dcad4e49): 4925 bytes on disk
I20240502 14:16:37.865514 26626 maintenance_manager.cc:643] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: UndoDeltaBlockGCOp(c41522c60f3942aa9f28f514dcad4e49) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20240502 14:16:37.867897 26735 maintenance_manager.cc:419] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Scheduling UndoDeltaBlockGCOp(1fb1ef75829641a89c4cac539f431d80): 4586 bytes on disk
I20240502 14:16:37.868513 26626 maintenance_manager.cc:643] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: UndoDeltaBlockGCOp(1fb1ef75829641a89c4cac539f431d80) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
I20240502 14:16:37.931130 26816 consensus_queue.cc:237] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:16:37.933646 29309 raft_consensus.cc:1261] T 40aa23be0caa427e89e8fdaf905dc052 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:37.933900 26686 raft_consensus.cc:1261] T 40aa23be0caa427e89e8fdaf905dc052 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:16:37.934228 10085 consensus_queue.cc:1035] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:37.934501 9547 consensus_queue.cc:1035] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:37.937312 9547 raft_consensus.cc:2932] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:37.937942 26686 raft_consensus.cc:2932] T 40aa23be0caa427e89e8fdaf905dc052 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:37.938184 29309 raft_consensus.cc:2932] T 40aa23be0caa427e89e8fdaf905dc052 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:37.945523 26284 catalog_manager.cc:5484] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: config changed from index 4 to 5, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) evicted. New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:37.960649 26406 tablet_service.cc:1514] Processing DeleteTablet for tablet 40aa23be0caa427e89e8fdaf905dc052 with delete_type TABLET_DATA_TOMBSTONED (TS 89af7f423cf44717a08d6b142f0ed49d not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:37422
I20240502 14:16:37.975764 10135 tablet_replica.cc:330] T 40aa23be0caa427e89e8fdaf905dc052 P 89af7f423cf44717a08d6b142f0ed49d: stopping tablet replica
I20240502 14:16:37.976123 10135 raft_consensus.cc:2229] T 40aa23be0caa427e89e8fdaf905dc052 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:37.976644 10135 raft_consensus.cc:2258] T 40aa23be0caa427e89e8fdaf905dc052 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:37.981177 10135 ts_tablet_manager.cc:1875] T 40aa23be0caa427e89e8fdaf905dc052 P 89af7f423cf44717a08d6b142f0ed49d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:37.989737 10135 ts_tablet_manager.cc:1888] T 40aa23be0caa427e89e8fdaf905dc052 P 89af7f423cf44717a08d6b142f0ed49d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:16:37.989982 10135 log.cc:1196] T 40aa23be0caa427e89e8fdaf905dc052 P 89af7f423cf44717a08d6b142f0ed49d: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/40aa23be0caa427e89e8fdaf905dc052
I20240502 14:16:37.991179 26271 catalog_manager.cc:4830] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): tablet 40aa23be0caa427e89e8fdaf905dc052 (table rebalance_test_table_17 [id=796786e80d834485b2cee5a25161aa97]) successfully deleted
I20240502 14:16:38.072232 26427 consensus_queue.cc:237] T d344462241604470a48a443b396177a7 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:16:38.074579 29174 raft_consensus.cc:1261] T d344462241604470a48a443b396177a7 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:38.074707 29309 raft_consensus.cc:1261] T d344462241604470a48a443b396177a7 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:16:38.074923 26362 consensus_peers.cc:485] T d344462241604470a48a443b396177a7 P 89af7f423cf44717a08d6b142f0ed49d -> Peer ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): Couldn't send request to peer ba187e38a2754d60a12de9ac4856430a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d344462241604470a48a443b396177a7. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:38.075227 10163 consensus_queue.cc:1035] T d344462241604470a48a443b396177a7 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:38.075515 9440 consensus_queue.cc:1035] T d344462241604470a48a443b396177a7 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:38.102013 10163 raft_consensus.cc:2932] T d344462241604470a48a443b396177a7 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:16:38.107342 29174 raft_consensus.cc:2932] T d344462241604470a48a443b396177a7 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:16:38.110110 26283 catalog_manager.cc:5484] T d344462241604470a48a443b396177a7 P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:16:38.123535 29309 raft_consensus.cc:2932] T d344462241604470a48a443b396177a7 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:16:38.519675 10216 ts_tablet_manager.cc:902] T d344462241604470a48a443b396177a7 P ba187e38a2754d60a12de9ac4856430a: Initiating tablet copy from peer 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:16:38.550558 10216 tablet_copy_client.cc:308] T d344462241604470a48a443b396177a7 P ba187e38a2754d60a12de9ac4856430a: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.193:37985
I20240502 14:16:38.551795 26447 tablet_copy_service.cc:140] P 89af7f423cf44717a08d6b142f0ed49d: Received BeginTabletCopySession request for tablet d344462241604470a48a443b396177a7 from peer ba187e38a2754d60a12de9ac4856430a ({username='slave'} at 127.19.94.194:39089)
I20240502 14:16:38.551987 26447 tablet_copy_service.cc:161] P 89af7f423cf44717a08d6b142f0ed49d: Beginning new tablet copy session on tablet d344462241604470a48a443b396177a7 from peer ba187e38a2754d60a12de9ac4856430a at {username='slave'} at 127.19.94.194:39089: session id = ba187e38a2754d60a12de9ac4856430a-d344462241604470a48a443b396177a7
I20240502 14:16:38.556824 26447 tablet_copy_source_session.cc:205] T d344462241604470a48a443b396177a7 P 89af7f423cf44717a08d6b142f0ed49d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:38.563087 10216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d344462241604470a48a443b396177a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:38.566912 29155 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:38.572973 10216 tablet_copy_client.cc:783] T d344462241604470a48a443b396177a7 P ba187e38a2754d60a12de9ac4856430a: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:38.573415 10216 tablet_copy_client.cc:647] T d344462241604470a48a443b396177a7 P ba187e38a2754d60a12de9ac4856430a: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:38.575222 10216 tablet_copy_client.cc:523] T d344462241604470a48a443b396177a7 P ba187e38a2754d60a12de9ac4856430a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:38.581271 10216 tablet_bootstrap.cc:492] T d344462241604470a48a443b396177a7 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:16:38.597354 10216 tablet_bootstrap.cc:492] T d344462241604470a48a443b396177a7 P ba187e38a2754d60a12de9ac4856430a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:38.607777 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:38.608711 26406 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 14:16:38.639364 29378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.198:43659, user_credentials={real_user=slave}} blocked reactor thread for 57283.7us
I20240502 14:16:38.644076 10216 tablet_bootstrap.cc:492] T d344462241604470a48a443b396177a7 P ba187e38a2754d60a12de9ac4856430a: Bootstrap complete.
I20240502 14:16:38.644834 10216 ts_tablet_manager.cc:1367] T d344462241604470a48a443b396177a7 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.064s user 0.011s sys 0.002s
I20240502 14:16:38.645733 10216 raft_consensus.cc:348] T d344462241604470a48a443b396177a7 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:16:38.649152 10216 raft_consensus.cc:729] T d344462241604470a48a443b396177a7 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:16:38.652076 10216 consensus_queue.cc:260] T d344462241604470a48a443b396177a7 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:16:38.653708 10216 ts_tablet_manager.cc:1398] T d344462241604470a48a443b396177a7 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 14:16:38.657590 26447 tablet_copy_service.cc:342] P 89af7f423cf44717a08d6b142f0ed49d: Request end of tablet copy session ba187e38a2754d60a12de9ac4856430a-d344462241604470a48a443b396177a7 received from {username='slave'} at 127.19.94.194:39089
I20240502 14:16:38.657732 26447 tablet_copy_service.cc:433] P 89af7f423cf44717a08d6b142f0ed49d: ending tablet copy session ba187e38a2754d60a12de9ac4856430a-d344462241604470a48a443b396177a7 on tablet d344462241604470a48a443b396177a7 with peer ba187e38a2754d60a12de9ac4856430a
I20240502 14:16:38.670715 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:38.679962 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:38.687863 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:38.741225 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:38.794456 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:38.923821 26557 raft_consensus.cc:1203] T d344462241604470a48a443b396177a7 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:16:39.344219 29155 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:39.363723 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:39.385254 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:39.405810 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:39.439074 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:39.439198 26406 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:39.478670 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:39.543296 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:39.720322 29175 consensus_queue.cc:237] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:16:39.723165 29309 raft_consensus.cc:1261] T 5e66bc4ba1f84b50884d6492edb1e0d2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 2 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:16:39.723294 29443 raft_consensus.cc:1261] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 68a3c2b17ed94e06aba66ed4178132ab [term 2 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:16:39.723809 10090 consensus_queue.cc:1035] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:39.724344 10090 consensus_queue.cc:1035] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:39.738199 10105 raft_consensus.cc:2932] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:39.741123 29443 raft_consensus.cc:2932] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 68a3c2b17ed94e06aba66ed4178132ab [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:39.751011 26283 catalog_manager.cc:5484] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 68a3c2b17ed94e06aba66ed4178132ab reported cstate change: config changed from index 2 to 4, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) evicted. New cstate: current_term: 2 leader_uuid: "1cb67158133e4648a6eceba32d4dbba3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:39.765698 29309 raft_consensus.cc:2932] T 5e66bc4ba1f84b50884d6492edb1e0d2 P c1100e16b3b9495ca61d4f95e89b6a89 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:39.774906 26865 maintenance_manager.cc:419] P e4cfb8fc5a474177af6848a6efa13a27: Scheduling FlushMRSOp(533b42de2de14abc96c00a928d5deed1): perf score=0.033341
I20240502 14:16:39.782989 26538 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e66bc4ba1f84b50884d6492edb1e0d2 with delete_type TABLET_DATA_TOMBSTONED (TS ba187e38a2754d60a12de9ac4856430a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51696
I20240502 14:16:39.807462 10362 tablet_replica.cc:330] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a: stopping tablet replica
I20240502 14:16:39.807777 10362 raft_consensus.cc:2229] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:39.808053 10362 raft_consensus.cc:2258] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:39.808889 10362 ts_tablet_manager.cc:1875] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:39.816779 10362 ts_tablet_manager.cc:1888] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240502 14:16:39.817050 10362 log.cc:1196] T 5e66bc4ba1f84b50884d6492edb1e0d2 P ba187e38a2754d60a12de9ac4856430a: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/5e66bc4ba1f84b50884d6492edb1e0d2
I20240502 14:16:39.818322 26271 catalog_manager.cc:4830] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): tablet 5e66bc4ba1f84b50884d6492edb1e0d2 (table rebalancer_extra_table_50 [id=183571cb1d8d40aaa35266ced8ac1c76]) successfully deleted
I20240502 14:16:39.819900 26604 maintenance_manager.cc:419] P ba187e38a2754d60a12de9ac4856430a: Scheduling FlushMRSOp(62218b2a597e4fb188ced6d2b9a83a59): perf score=0.033340
I20240502 14:16:39.844687 26756 maintenance_manager.cc:643] P e4cfb8fc5a474177af6848a6efa13a27: FlushMRSOp(533b42de2de14abc96c00a928d5deed1) complete. Timing: real 0.069s user 0.046s sys 0.006s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":529,"dirs.run_cpu_time_us":243,"dirs.run_wall_time_us":5746,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":800,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":7139,"threads_started":2,"wal-append.queue_time_us":6561}
I20240502 14:16:39.848184 26865 maintenance_manager.cc:419] P e4cfb8fc5a474177af6848a6efa13a27: Scheduling UndoDeltaBlockGCOp(533b42de2de14abc96c00a928d5deed1): 4838 bytes on disk
I20240502 14:16:39.848982 26756 maintenance_manager.cc:643] P e4cfb8fc5a474177af6848a6efa13a27: UndoDeltaBlockGCOp(533b42de2de14abc96c00a928d5deed1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4}
I20240502 14:16:39.862628 26865 maintenance_manager.cc:419] P e4cfb8fc5a474177af6848a6efa13a27: Scheduling FlushMRSOp(6b4745b1153b4acd999ac3a5c137ff53): perf score=0.033353
I20240502 14:16:39.879309 26473 maintenance_manager.cc:419] P 89af7f423cf44717a08d6b142f0ed49d: Scheduling FlushMRSOp(6b4745b1153b4acd999ac3a5c137ff53): perf score=0.033361
I20240502 14:16:39.890522 26735 maintenance_manager.cc:419] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Scheduling FlushMRSOp(62218b2a597e4fb188ced6d2b9a83a59): perf score=0.033352
I20240502 14:16:39.942935 26756 maintenance_manager.cc:643] P e4cfb8fc5a474177af6848a6efa13a27: FlushMRSOp(6b4745b1153b4acd999ac3a5c137ff53) complete. Timing: real 0.080s user 0.034s sys 0.032s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":58,"dirs.run_cpu_time_us":217,"dirs.run_wall_time_us":1665,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":804,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":15506,"threads_started":1,"wal-append.queue_time_us":15610}
I20240502 14:16:39.946262 26865 maintenance_manager.cc:419] P e4cfb8fc5a474177af6848a6efa13a27: Scheduling UndoDeltaBlockGCOp(6b4745b1153b4acd999ac3a5c137ff53): 4923 bytes on disk
I20240502 14:16:39.947117 26495 maintenance_manager.cc:643] P ba187e38a2754d60a12de9ac4856430a: FlushMRSOp(62218b2a597e4fb188ced6d2b9a83a59) complete. Timing: real 0.127s user 0.065s sys 0.014s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":23757,"dirs.run_cpu_time_us":258,"dirs.run_wall_time_us":8226,"drs_written":1,"lbm_read_time_us":131,"lbm_reads_lt_1ms":4,"lbm_write_time_us":927,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":28911,"threads_started":2,"wal-append.queue_time_us":4809}
I20240502 14:16:39.950492 26604 maintenance_manager.cc:419] P ba187e38a2754d60a12de9ac4856430a: Scheduling UndoDeltaBlockGCOp(62218b2a597e4fb188ced6d2b9a83a59): 4582 bytes on disk
I20240502 14:16:39.954890 26756 maintenance_manager.cc:643] P e4cfb8fc5a474177af6848a6efa13a27: UndoDeltaBlockGCOp(6b4745b1153b4acd999ac3a5c137ff53) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
I20240502 14:16:39.956354 26495 maintenance_manager.cc:643] P ba187e38a2754d60a12de9ac4856430a: UndoDeltaBlockGCOp(62218b2a597e4fb188ced6d2b9a83a59) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4}
I20240502 14:16:39.982700 26626 maintenance_manager.cc:643] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: FlushMRSOp(62218b2a597e4fb188ced6d2b9a83a59) complete. Timing: real 0.091s user 0.039s sys 0.019s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":6377,"dirs.run_cpu_time_us":215,"dirs.run_wall_time_us":8842,"drs_written":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4,"lbm_write_time_us":897,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":20941,"threads_started":2,"wal-append.queue_time_us":14502}
I20240502 14:16:39.985965 26735 maintenance_manager.cc:419] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Scheduling UndoDeltaBlockGCOp(62218b2a597e4fb188ced6d2b9a83a59): 4582 bytes on disk
I20240502 14:16:39.986833 26626 maintenance_manager.cc:643] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: UndoDeltaBlockGCOp(62218b2a597e4fb188ced6d2b9a83a59) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":119,"lbm_reads_lt_1ms":4}
I20240502 14:16:39.998785 26364 maintenance_manager.cc:643] P 89af7f423cf44717a08d6b142f0ed49d: FlushMRSOp(6b4745b1153b4acd999ac3a5c137ff53) complete. Timing: real 0.119s user 0.045s sys 0.009s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":2304,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":4571,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2001,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":2674,"threads_started":2,"wal-append.queue_time_us":361}
I20240502 14:16:40.001734 26473 maintenance_manager.cc:419] P 89af7f423cf44717a08d6b142f0ed49d: Scheduling UndoDeltaBlockGCOp(6b4745b1153b4acd999ac3a5c137ff53): 4923 bytes on disk
I20240502 14:16:40.003144 26364 maintenance_manager.cc:643] P 89af7f423cf44717a08d6b142f0ed49d: UndoDeltaBlockGCOp(6b4745b1153b4acd999ac3a5c137ff53) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
I20240502 14:16:40.006060 26473 maintenance_manager.cc:419] P 89af7f423cf44717a08d6b142f0ed49d: Scheduling FlushMRSOp(533b42de2de14abc96c00a928d5deed1): perf score=0.033386
I20240502 14:16:40.033931 29574 consensus_queue.cc:237] T fc6401f0ab8c485399325df7c8dd96ad P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:16:40.038347 26556 raft_consensus.cc:1261] T fc6401f0ab8c485399325df7c8dd96ad P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:40.039426 9576 consensus_queue.cc:1035] T fc6401f0ab8c485399325df7c8dd96ad P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:40.040343 29443 raft_consensus.cc:1261] T fc6401f0ab8c485399325df7c8dd96ad P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:16:40.040760 29511 consensus_peers.cc:485] T fc6401f0ab8c485399325df7c8dd96ad P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): Couldn't send request to peer 69ab78b2a1b74d2bbf2cdeafa0a4af23. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc6401f0ab8c485399325df7c8dd96ad. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:40.041028 10319 consensus_queue.cc:1035] T fc6401f0ab8c485399325df7c8dd96ad P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:40.052204 10319 raft_consensus.cc:2932] T fc6401f0ab8c485399325df7c8dd96ad P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:40.064270 26284 catalog_manager.cc:5484] T fc6401f0ab8c485399325df7c8dd96ad P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:40.069118 26556 raft_consensus.cc:2932] T fc6401f0ab8c485399325df7c8dd96ad P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:40.075682 29443 raft_consensus.cc:2932] T fc6401f0ab8c485399325df7c8dd96ad P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:40.126377 26364 maintenance_manager.cc:643] P 89af7f423cf44717a08d6b142f0ed49d: FlushMRSOp(533b42de2de14abc96c00a928d5deed1) complete. Timing: real 0.120s user 0.053s sys 0.018s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":43,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":1459,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":958,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":31739,"threads_started":1,"wal-append.queue_time_us":28058}
I20240502 14:16:40.128911 26473 maintenance_manager.cc:419] P 89af7f423cf44717a08d6b142f0ed49d: Scheduling UndoDeltaBlockGCOp(533b42de2de14abc96c00a928d5deed1): 4838 bytes on disk
I20240502 14:16:40.129680 26364 maintenance_manager.cc:643] P 89af7f423cf44717a08d6b142f0ed49d: UndoDeltaBlockGCOp(533b42de2de14abc96c00a928d5deed1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4}
I20240502 14:16:40.199564 29443 consensus_queue.cc:237] T eb065120d1304a2a81ce932934b911f3 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
W20240502 14:16:40.213730 29378 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): Couldn't send request to peer 89af7f423cf44717a08d6b142f0ed49d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb065120d1304a2a81ce932934b911f3. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:40.214486 29307 raft_consensus.cc:1261] T eb065120d1304a2a81ce932934b911f3 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:40.217422 9901 consensus_queue.cc:1035] T eb065120d1304a2a81ce932934b911f3 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:40.218937 29175 raft_consensus.cc:1261] T eb065120d1304a2a81ce932934b911f3 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:40.222088 9901 consensus_queue.cc:1035] T eb065120d1304a2a81ce932934b911f3 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:40.262235 10232 raft_consensus.cc:2932] T eb065120d1304a2a81ce932934b911f3 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:40.262905 29175 raft_consensus.cc:2932] T eb065120d1304a2a81ce932934b911f3 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:40.268056 29307 raft_consensus.cc:2932] T eb065120d1304a2a81ce932934b911f3 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:40.270437 26284 catalog_manager.cc:5484] T eb065120d1304a2a81ce932934b911f3 P 68a3c2b17ed94e06aba66ed4178132ab reported cstate change: config changed from index -1 to 2, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New cstate: current_term: 1 leader_uuid: "68a3c2b17ed94e06aba66ed4178132ab" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:40.321463 26427 consensus_queue.cc:237] T a04b8c49626e491b845d9c81142ac659 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:16:40.324338 29443 raft_consensus.cc:1261] T a04b8c49626e491b845d9c81142ac659 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:40.324452 26815 raft_consensus.cc:1261] T a04b8c49626e491b845d9c81142ac659 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:40.325085 10163 consensus_queue.cc:1035] T a04b8c49626e491b845d9c81142ac659 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:40.325438 9440 consensus_queue.cc:1035] T a04b8c49626e491b845d9c81142ac659 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:16:40.334751 26363 consensus_peers.cc:485] T a04b8c49626e491b845d9c81142ac659 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): Couldn't send request to peer 69ab78b2a1b74d2bbf2cdeafa0a4af23. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a04b8c49626e491b845d9c81142ac659. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:40.342391 10163 raft_consensus.cc:2932] T a04b8c49626e491b845d9c81142ac659 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:40.343304 29443 raft_consensus.cc:2932] T a04b8c49626e491b845d9c81142ac659 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:40.350039 26815 raft_consensus.cc:2932] T a04b8c49626e491b845d9c81142ac659 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:40.350381 26285 catalog_manager.cc:5484] T a04b8c49626e491b845d9c81142ac659 P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:40.409677 29175 consensus_queue.cc:237] T 65b7c6aa6aac45079adcff73f465acdb P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:16:40.413334 26686 raft_consensus.cc:1261] T 65b7c6aa6aac45079adcff73f465acdb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:40.413914 29307 raft_consensus.cc:1261] T 65b7c6aa6aac45079adcff73f465acdb P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:16:40.414698 29111 consensus_peers.cc:485] T 65b7c6aa6aac45079adcff73f465acdb P 1cb67158133e4648a6eceba32d4dbba3 -> Peer ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): Couldn't send request to peer ba187e38a2754d60a12de9ac4856430a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 65b7c6aa6aac45079adcff73f465acdb. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:40.415024 10090 consensus_queue.cc:1035] T 65b7c6aa6aac45079adcff73f465acdb P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:40.415612 10090 consensus_queue.cc:1035] T 65b7c6aa6aac45079adcff73f465acdb P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:40.419582 10090 raft_consensus.cc:2932] T 65b7c6aa6aac45079adcff73f465acdb P 1cb67158133e4648a6eceba32d4dbba3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:16:40.420640 26687 raft_consensus.cc:2932] T 65b7c6aa6aac45079adcff73f465acdb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:16:40.428753 26284 catalog_manager.cc:5484] T 65b7c6aa6aac45079adcff73f465acdb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New cstate: current_term: 1 leader_uuid: "1cb67158133e4648a6eceba32d4dbba3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:16:40.435822 29307 raft_consensus.cc:2932] T 65b7c6aa6aac45079adcff73f465acdb P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:16:40.560798 29574 consensus_queue.cc:237] T a262ac98859b4208b70573662cb898cb P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:16:40.563323 29175 raft_consensus.cc:1261] T a262ac98859b4208b70573662cb898cb P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:16:40.563382 29511 consensus_peers.cc:485] T a262ac98859b4208b70573662cb898cb P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): Couldn't send request to peer 69ab78b2a1b74d2bbf2cdeafa0a4af23. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a262ac98859b4208b70573662cb898cb. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:40.563912 29438 raft_consensus.cc:1261] T a262ac98859b4208b70573662cb898cb P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:40.563910 10399 consensus_queue.cc:1035] T a262ac98859b4208b70573662cb898cb P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:40.564441 10400 consensus_queue.cc:1035] T a262ac98859b4208b70573662cb898cb P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:40.569700 10399 raft_consensus.cc:2932] T a262ac98859b4208b70573662cb898cb P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:40.570434 29175 raft_consensus.cc:2932] T a262ac98859b4208b70573662cb898cb P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:40.578275 26284 catalog_manager.cc:5484] T a262ac98859b4208b70573662cb898cb P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:40.580068 29438 raft_consensus.cc:2932] T a262ac98859b4208b70573662cb898cb P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:40.655956 10497 ts_tablet_manager.cc:902] T fc6401f0ab8c485399325df7c8dd96ad P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Initiating tablet copy from peer 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:16:40.656838 26816 consensus_queue.cc:237] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:16:40.660252 29438 raft_consensus.cc:1261] T dc3740d0671d42bc90eb9baeb02f5b0a P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:40.660666 29574 raft_consensus.cc:1261] T dc3740d0671d42bc90eb9baeb02f5b0a P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:40.660867 10128 consensus_queue.cc:1035] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:16:40.661803 26755 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): Couldn't send request to peer 69ab78b2a1b74d2bbf2cdeafa0a4af23. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc3740d0671d42bc90eb9baeb02f5b0a. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:40.662941 10128 consensus_queue.cc:1035] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:40.669204 10497 tablet_copy_client.cc:308] T fc6401f0ab8c485399325df7c8dd96ad P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.200:46097
I20240502 14:16:40.669926 10303 raft_consensus.cc:2932] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:40.670303 29594 tablet_copy_service.cc:140] P 4092902b3e674857a0147f1b4f2cdb86: Received BeginTabletCopySession request for tablet fc6401f0ab8c485399325df7c8dd96ad from peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 ({username='slave'} at 127.19.94.195:39709)
I20240502 14:16:40.670449 29594 tablet_copy_service.cc:161] P 4092902b3e674857a0147f1b4f2cdb86: Beginning new tablet copy session on tablet fc6401f0ab8c485399325df7c8dd96ad from peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 at {username='slave'} at 127.19.94.195:39709: session id = 69ab78b2a1b74d2bbf2cdeafa0a4af23-fc6401f0ab8c485399325df7c8dd96ad
I20240502 14:16:40.670552 29574 raft_consensus.cc:2932] T dc3740d0671d42bc90eb9baeb02f5b0a P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:40.676267 29594 tablet_copy_source_session.cc:205] T fc6401f0ab8c485399325df7c8dd96ad P 4092902b3e674857a0147f1b4f2cdb86: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:40.676311 29443 raft_consensus.cc:2932] T dc3740d0671d42bc90eb9baeb02f5b0a P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:40.683303 10497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc6401f0ab8c485399325df7c8dd96ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:40.691416 26285 catalog_manager.cc:5484] T dc3740d0671d42bc90eb9baeb02f5b0a P 68a3c2b17ed94e06aba66ed4178132ab reported cstate change: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:40.695574 10497 tablet_copy_client.cc:783] T fc6401f0ab8c485399325df7c8dd96ad P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:40.695902 10497 tablet_copy_client.cc:647] T fc6401f0ab8c485399325df7c8dd96ad P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:40.699555 10497 tablet_copy_client.cc:523] T fc6401f0ab8c485399325df7c8dd96ad P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:40.703737 10497 tablet_bootstrap.cc:492] T fc6401f0ab8c485399325df7c8dd96ad P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:16:40.711472 10497 tablet_bootstrap.cc:492] T fc6401f0ab8c485399325df7c8dd96ad P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:40.712141 10497 tablet_bootstrap.cc:492] T fc6401f0ab8c485399325df7c8dd96ad P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap complete.
I20240502 14:16:40.712451 10497 ts_tablet_manager.cc:1367] T fc6401f0ab8c485399325df7c8dd96ad P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20240502 14:16:40.713747 10497 raft_consensus.cc:348] T fc6401f0ab8c485399325df7c8dd96ad P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:16:40.713930 10497 raft_consensus.cc:729] T fc6401f0ab8c485399325df7c8dd96ad P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:16:40.714191 10497 consensus_queue.cc:260] T fc6401f0ab8c485399325df7c8dd96ad P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:16:40.719964 10497 ts_tablet_manager.cc:1398] T fc6401f0ab8c485399325df7c8dd96ad P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:16:40.720904 29594 tablet_copy_service.cc:342] P 4092902b3e674857a0147f1b4f2cdb86: Request end of tablet copy session 69ab78b2a1b74d2bbf2cdeafa0a4af23-fc6401f0ab8c485399325df7c8dd96ad received from {username='slave'} at 127.19.94.195:39709
I20240502 14:16:40.721067 29594 tablet_copy_service.cc:433] P 4092902b3e674857a0147f1b4f2cdb86: ending tablet copy session 69ab78b2a1b74d2bbf2cdeafa0a4af23-fc6401f0ab8c485399325df7c8dd96ad on tablet fc6401f0ab8c485399325df7c8dd96ad with peer 69ab78b2a1b74d2bbf2cdeafa0a4af23
I20240502 14:16:40.754638 10497 ts_tablet_manager.cc:902] T a04b8c49626e491b845d9c81142ac659 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Initiating tablet copy from peer 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:16:40.777223 29574 consensus_queue.cc:237] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:16:40.787519 29309 raft_consensus.cc:1261] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:40.788024 26425 raft_consensus.cc:1261] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:16:40.788416 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2dc3b47b4ab145ccb9efbf1b5740c7f6. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:40.788551 10525 consensus_queue.cc:1035] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:40.788839 10400 consensus_queue.cc:1035] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:40.810094 10525 raft_consensus.cc:2932] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:40.813232 26425 raft_consensus.cc:2932] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:40.816947 29309 raft_consensus.cc:2932] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:40.826625 26284 catalog_manager.cc:5484] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P c1100e16b3b9495ca61d4f95e89b6a89 reported cstate change: config changed from index -1 to 2, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:40.833195 10529 ts_tablet_manager.cc:902] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d: Initiating tablet copy from peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:16:40.842280 10497 tablet_copy_client.cc:308] T a04b8c49626e491b845d9c81142ac659 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.193:37985
I20240502 14:16:40.843430 26447 tablet_copy_service.cc:140] P 89af7f423cf44717a08d6b142f0ed49d: Received BeginTabletCopySession request for tablet a04b8c49626e491b845d9c81142ac659 from peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 ({username='slave'} at 127.19.94.195:34637)
I20240502 14:16:40.843603 26447 tablet_copy_service.cc:161] P 89af7f423cf44717a08d6b142f0ed49d: Beginning new tablet copy session on tablet a04b8c49626e491b845d9c81142ac659 from peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 at {username='slave'} at 127.19.94.195:34637: session id = 69ab78b2a1b74d2bbf2cdeafa0a4af23-a04b8c49626e491b845d9c81142ac659
I20240502 14:16:40.846158 10529 tablet_copy_client.cc:308] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.199:41781
I20240502 14:16:40.848572 29463 tablet_copy_service.cc:140] P 68a3c2b17ed94e06aba66ed4178132ab: Received BeginTabletCopySession request for tablet eb065120d1304a2a81ce932934b911f3 from peer 89af7f423cf44717a08d6b142f0ed49d ({username='slave'} at 127.19.94.193:53537)
I20240502 14:16:40.848774 29463 tablet_copy_service.cc:161] P 68a3c2b17ed94e06aba66ed4178132ab: Beginning new tablet copy session on tablet eb065120d1304a2a81ce932934b911f3 from peer 89af7f423cf44717a08d6b142f0ed49d at {username='slave'} at 127.19.94.193:53537: session id = 89af7f423cf44717a08d6b142f0ed49d-eb065120d1304a2a81ce932934b911f3
I20240502 14:16:40.851878 29463 tablet_copy_source_session.cc:205] T eb065120d1304a2a81ce932934b911f3 P 68a3c2b17ed94e06aba66ed4178132ab: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:40.848219 26447 tablet_copy_source_session.cc:205] T a04b8c49626e491b845d9c81142ac659 P 89af7f423cf44717a08d6b142f0ed49d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:40.854902 10529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb065120d1304a2a81ce932934b911f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:40.855962 10497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a04b8c49626e491b845d9c81142ac659. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:40.862216 10497 tablet_copy_client.cc:783] T a04b8c49626e491b845d9c81142ac659 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:40.862450 10497 tablet_copy_client.cc:647] T a04b8c49626e491b845d9c81142ac659 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:40.864127 10497 tablet_copy_client.cc:523] T a04b8c49626e491b845d9c81142ac659 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:40.867662 10497 tablet_bootstrap.cc:492] T a04b8c49626e491b845d9c81142ac659 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:16:40.867997 10529 tablet_copy_client.cc:783] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:40.868295 10529 tablet_copy_client.cc:647] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:40.870231 10529 tablet_copy_client.cc:523] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:40.873646 10497 tablet_bootstrap.cc:492] T a04b8c49626e491b845d9c81142ac659 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:40.873907 10529 tablet_bootstrap.cc:492] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:16:40.874001 10497 tablet_bootstrap.cc:492] T a04b8c49626e491b845d9c81142ac659 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap complete.
I20240502 14:16:40.874212 10497 ts_tablet_manager.cc:1367] T a04b8c49626e491b845d9c81142ac659 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240502 14:16:40.874804 10497 raft_consensus.cc:348] T a04b8c49626e491b845d9c81142ac659 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:16:40.874960 10497 raft_consensus.cc:729] T a04b8c49626e491b845d9c81142ac659 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:16:40.875156 10497 consensus_queue.cc:260] T a04b8c49626e491b845d9c81142ac659 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:16:40.875543 10497 ts_tablet_manager.cc:1398] T a04b8c49626e491b845d9c81142ac659 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:16:40.876879 26447 tablet_copy_service.cc:342] P 89af7f423cf44717a08d6b142f0ed49d: Request end of tablet copy session 69ab78b2a1b74d2bbf2cdeafa0a4af23-a04b8c49626e491b845d9c81142ac659 received from {username='slave'} at 127.19.94.195:34637
I20240502 14:16:40.876991 26447 tablet_copy_service.cc:433] P 89af7f423cf44717a08d6b142f0ed49d: ending tablet copy session 69ab78b2a1b74d2bbf2cdeafa0a4af23-a04b8c49626e491b845d9c81142ac659 on tablet a04b8c49626e491b845d9c81142ac659 with peer 69ab78b2a1b74d2bbf2cdeafa0a4af23
I20240502 14:16:40.886198 10529 tablet_bootstrap.cc:492] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:40.886610 10529 tablet_bootstrap.cc:492] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap complete.
I20240502 14:16:40.886833 10529 ts_tablet_manager.cc:1367] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.002s
I20240502 14:16:40.887521 10529 raft_consensus.cc:348] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:40.887742 10529 raft_consensus.cc:729] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:16:40.887987 10529 consensus_queue.cc:260] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:40.888849 10529 ts_tablet_manager.cc:1398] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
I20240502 14:16:40.889712 29463 tablet_copy_service.cc:342] P 68a3c2b17ed94e06aba66ed4178132ab: Request end of tablet copy session 89af7f423cf44717a08d6b142f0ed49d-eb065120d1304a2a81ce932934b911f3 received from {username='slave'} at 127.19.94.193:53537
I20240502 14:16:40.889853 29463 tablet_copy_service.cc:433] P 68a3c2b17ed94e06aba66ed4178132ab: ending tablet copy session 89af7f423cf44717a08d6b142f0ed49d-eb065120d1304a2a81ce932934b911f3 on tablet eb065120d1304a2a81ce932934b911f3 with peer 89af7f423cf44717a08d6b142f0ed49d
I20240502 14:16:41.059172 10577 ts_tablet_manager.cc:902] T 65b7c6aa6aac45079adcff73f465acdb P ba187e38a2754d60a12de9ac4856430a: Initiating tablet copy from peer 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:16:41.110689 10497 ts_tablet_manager.cc:902] T a262ac98859b4208b70573662cb898cb P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Initiating tablet copy from peer 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:16:41.145022 26687 raft_consensus.cc:1203] T a04b8c49626e491b845d9c81142ac659 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:16:41.146750 10497 tablet_copy_client.cc:308] T a262ac98859b4208b70573662cb898cb P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.200:46097
I20240502 14:16:41.148308 29594 tablet_copy_service.cc:140] P 4092902b3e674857a0147f1b4f2cdb86: Received BeginTabletCopySession request for tablet a262ac98859b4208b70573662cb898cb from peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 ({username='slave'} at 127.19.94.195:39709)
I20240502 14:16:41.148633 29594 tablet_copy_service.cc:161] P 4092902b3e674857a0147f1b4f2cdb86: Beginning new tablet copy session on tablet a262ac98859b4208b70573662cb898cb from peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 at {username='slave'} at 127.19.94.195:39709: session id = 69ab78b2a1b74d2bbf2cdeafa0a4af23-a262ac98859b4208b70573662cb898cb
I20240502 14:16:41.194252 29594 tablet_copy_source_session.cc:205] T a262ac98859b4208b70573662cb898cb P 4092902b3e674857a0147f1b4f2cdb86: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:41.196220 10497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a262ac98859b4208b70573662cb898cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:41.203382 10497 tablet_copy_client.cc:783] T a262ac98859b4208b70573662cb898cb P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:41.203629 10497 tablet_copy_client.cc:647] T a262ac98859b4208b70573662cb898cb P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:41.207998 10497 tablet_copy_client.cc:523] T a262ac98859b4208b70573662cb898cb P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:41.214907 10497 tablet_bootstrap.cc:492] T a262ac98859b4208b70573662cb898cb P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:16:41.244779 10589 ts_tablet_manager.cc:902] T dc3740d0671d42bc90eb9baeb02f5b0a P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Initiating tablet copy from peer e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:16:41.255617 26687 raft_consensus.cc:1203] T fc6401f0ab8c485399325df7c8dd96ad P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:16:41.279767 10497 tablet_bootstrap.cc:492] T a262ac98859b4208b70573662cb898cb P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:41.286592 10577 tablet_copy_client.cc:308] T 65b7c6aa6aac45079adcff73f465acdb P ba187e38a2754d60a12de9ac4856430a: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.197:38715
I20240502 14:16:41.287443 29196 tablet_copy_service.cc:140] P 1cb67158133e4648a6eceba32d4dbba3: Received BeginTabletCopySession request for tablet 65b7c6aa6aac45079adcff73f465acdb from peer ba187e38a2754d60a12de9ac4856430a ({username='slave'} at 127.19.94.194:50529)
I20240502 14:16:41.287595 29196 tablet_copy_service.cc:161] P 1cb67158133e4648a6eceba32d4dbba3: Beginning new tablet copy session on tablet 65b7c6aa6aac45079adcff73f465acdb from peer ba187e38a2754d60a12de9ac4856430a at {username='slave'} at 127.19.94.194:50529: session id = ba187e38a2754d60a12de9ac4856430a-65b7c6aa6aac45079adcff73f465acdb
I20240502 14:16:41.290553 29196 tablet_copy_source_session.cc:205] T 65b7c6aa6aac45079adcff73f465acdb P 1cb67158133e4648a6eceba32d4dbba3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:41.293280 10577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65b7c6aa6aac45079adcff73f465acdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:41.296273 10497 tablet_bootstrap.cc:492] T a262ac98859b4208b70573662cb898cb P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap complete.
I20240502 14:16:41.298305 10497 ts_tablet_manager.cc:1367] T a262ac98859b4208b70573662cb898cb P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.083s user 0.008s sys 0.007s
I20240502 14:16:41.299052 10497 raft_consensus.cc:348] T a262ac98859b4208b70573662cb898cb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:16:41.299293 10497 raft_consensus.cc:729] T a262ac98859b4208b70573662cb898cb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:16:41.299571 10497 consensus_queue.cc:260] T a262ac98859b4208b70573662cb898cb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:16:41.300879 10577 tablet_copy_client.cc:783] T 65b7c6aa6aac45079adcff73f465acdb P ba187e38a2754d60a12de9ac4856430a: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:41.301110 10577 tablet_copy_client.cc:647] T 65b7c6aa6aac45079adcff73f465acdb P ba187e38a2754d60a12de9ac4856430a: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:41.304777 10589 tablet_copy_client.cc:308] T dc3740d0671d42bc90eb9baeb02f5b0a P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.196:32843
I20240502 14:16:41.305660 26839 tablet_copy_service.cc:140] P e4cfb8fc5a474177af6848a6efa13a27: Received BeginTabletCopySession request for tablet dc3740d0671d42bc90eb9baeb02f5b0a from peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 ({username='slave'} at 127.19.94.195:42141)
I20240502 14:16:41.305903 26839 tablet_copy_service.cc:161] P e4cfb8fc5a474177af6848a6efa13a27: Beginning new tablet copy session on tablet dc3740d0671d42bc90eb9baeb02f5b0a from peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 at {username='slave'} at 127.19.94.195:42141: session id = 69ab78b2a1b74d2bbf2cdeafa0a4af23-dc3740d0671d42bc90eb9baeb02f5b0a
I20240502 14:16:41.315712 10577 tablet_copy_client.cc:523] T 65b7c6aa6aac45079adcff73f465acdb P ba187e38a2754d60a12de9ac4856430a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:41.316314 26839 tablet_copy_source_session.cc:205] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:41.317896 10589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc3740d0671d42bc90eb9baeb02f5b0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:41.319526 26425 raft_consensus.cc:1203] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:16:41.321794 10497 ts_tablet_manager.cc:1398] T a262ac98859b4208b70573662cb898cb P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20240502 14:16:41.327505 29594 tablet_copy_service.cc:342] P 4092902b3e674857a0147f1b4f2cdb86: Request end of tablet copy session 69ab78b2a1b74d2bbf2cdeafa0a4af23-a262ac98859b4208b70573662cb898cb received from {username='slave'} at 127.19.94.195:39709
I20240502 14:16:41.327692 29594 tablet_copy_service.cc:433] P 4092902b3e674857a0147f1b4f2cdb86: ending tablet copy session 69ab78b2a1b74d2bbf2cdeafa0a4af23-a262ac98859b4208b70573662cb898cb on tablet a262ac98859b4208b70573662cb898cb with peer 69ab78b2a1b74d2bbf2cdeafa0a4af23
I20240502 14:16:41.331367 10589 tablet_copy_client.cc:783] T dc3740d0671d42bc90eb9baeb02f5b0a P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:41.330906 10577 tablet_bootstrap.cc:492] T 65b7c6aa6aac45079adcff73f465acdb P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:16:41.331665 10589 tablet_copy_client.cc:647] T dc3740d0671d42bc90eb9baeb02f5b0a P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:41.335649 10589 tablet_copy_client.cc:523] T dc3740d0671d42bc90eb9baeb02f5b0a P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:41.343887 10577 tablet_bootstrap.cc:492] T 65b7c6aa6aac45079adcff73f465acdb P ba187e38a2754d60a12de9ac4856430a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:41.344378 10577 tablet_bootstrap.cc:492] T 65b7c6aa6aac45079adcff73f465acdb P ba187e38a2754d60a12de9ac4856430a: Bootstrap complete.
I20240502 14:16:41.344668 10577 ts_tablet_manager.cc:1367] T 65b7c6aa6aac45079adcff73f465acdb P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.000s
I20240502 14:16:41.345443 10577 raft_consensus.cc:348] T 65b7c6aa6aac45079adcff73f465acdb P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:16:41.345661 10577 raft_consensus.cc:729] T 65b7c6aa6aac45079adcff73f465acdb P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:16:41.345924 10577 consensus_queue.cc:260] T 65b7c6aa6aac45079adcff73f465acdb P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:16:41.346586 10577 ts_tablet_manager.cc:1398] T 65b7c6aa6aac45079adcff73f465acdb P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:16:41.348170 29196 tablet_copy_service.cc:342] P 1cb67158133e4648a6eceba32d4dbba3: Request end of tablet copy session ba187e38a2754d60a12de9ac4856430a-65b7c6aa6aac45079adcff73f465acdb received from {username='slave'} at 127.19.94.194:50529
I20240502 14:16:41.348285 29196 tablet_copy_service.cc:433] P 1cb67158133e4648a6eceba32d4dbba3: ending tablet copy session ba187e38a2754d60a12de9ac4856430a-65b7c6aa6aac45079adcff73f465acdb on tablet 65b7c6aa6aac45079adcff73f465acdb with peer ba187e38a2754d60a12de9ac4856430a
I20240502 14:16:41.357825 10589 tablet_bootstrap.cc:492] T dc3740d0671d42bc90eb9baeb02f5b0a P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:16:41.374502 10606 ts_tablet_manager.cc:902] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 68a3c2b17ed94e06aba66ed4178132ab: Initiating tablet copy from peer 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:16:41.376899 10606 tablet_copy_client.cc:308] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.200:46097
I20240502 14:16:41.377853 29594 tablet_copy_service.cc:140] P 4092902b3e674857a0147f1b4f2cdb86: Received BeginTabletCopySession request for tablet 2dc3b47b4ab145ccb9efbf1b5740c7f6 from peer 68a3c2b17ed94e06aba66ed4178132ab ({username='slave'} at 127.19.94.199:59271)
I20240502 14:16:41.378115 29594 tablet_copy_service.cc:161] P 4092902b3e674857a0147f1b4f2cdb86: Beginning new tablet copy session on tablet 2dc3b47b4ab145ccb9efbf1b5740c7f6 from peer 68a3c2b17ed94e06aba66ed4178132ab at {username='slave'} at 127.19.94.199:59271: session id = 68a3c2b17ed94e06aba66ed4178132ab-2dc3b47b4ab145ccb9efbf1b5740c7f6
I20240502 14:16:41.383656 29594 tablet_copy_source_session.cc:205] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:41.395426 10606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dc3b47b4ab145ccb9efbf1b5740c7f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:41.400182 29155 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:41.403417 10606 tablet_copy_client.cc:783] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:41.404790 10606 tablet_copy_client.cc:647] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:41.413489 10589 tablet_bootstrap.cc:492] T dc3740d0671d42bc90eb9baeb02f5b0a P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:41.414018 10589 tablet_bootstrap.cc:492] T dc3740d0671d42bc90eb9baeb02f5b0a P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap complete.
I20240502 14:16:41.414299 10589 ts_tablet_manager.cc:1367] T dc3740d0671d42bc90eb9baeb02f5b0a P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.057s user 0.009s sys 0.000s
I20240502 14:16:41.414969 10589 raft_consensus.cc:348] T dc3740d0671d42bc90eb9baeb02f5b0a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:16:41.415246 10589 raft_consensus.cc:729] T dc3740d0671d42bc90eb9baeb02f5b0a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:16:41.415500 10589 consensus_queue.cc:260] T dc3740d0671d42bc90eb9baeb02f5b0a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:16:41.416074 10589 ts_tablet_manager.cc:1398] T dc3740d0671d42bc90eb9baeb02f5b0a P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:16:41.416500 10606 tablet_copy_client.cc:523] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:41.419289 26839 tablet_copy_service.cc:342] P e4cfb8fc5a474177af6848a6efa13a27: Request end of tablet copy session 69ab78b2a1b74d2bbf2cdeafa0a4af23-dc3740d0671d42bc90eb9baeb02f5b0a received from {username='slave'} at 127.19.94.195:42141
I20240502 14:16:41.419466 26839 tablet_copy_service.cc:433] P e4cfb8fc5a474177af6848a6efa13a27: ending tablet copy session 69ab78b2a1b74d2bbf2cdeafa0a4af23-dc3740d0671d42bc90eb9baeb02f5b0a on tablet dc3740d0671d42bc90eb9baeb02f5b0a with peer 69ab78b2a1b74d2bbf2cdeafa0a4af23
I20240502 14:16:41.431062 10606 tablet_bootstrap.cc:492] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:16:41.481439 10606 tablet_bootstrap.cc:492] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:41.481873 10606 tablet_bootstrap.cc:492] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap complete.
I20240502 14:16:41.482118 10606 ts_tablet_manager.cc:1367] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.051s user 0.004s sys 0.004s
I20240502 14:16:41.486595 10606 raft_consensus.cc:348] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:16:41.492952 10606 raft_consensus.cc:729] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:16:41.493233 10606 consensus_queue.cc:260] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:16:41.493865 10606 ts_tablet_manager.cc:1398] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20240502 14:16:41.496354 29594 tablet_copy_service.cc:342] P 4092902b3e674857a0147f1b4f2cdb86: Request end of tablet copy session 68a3c2b17ed94e06aba66ed4178132ab-2dc3b47b4ab145ccb9efbf1b5740c7f6 received from {username='slave'} at 127.19.94.199:59271
I20240502 14:16:41.496749 29594 tablet_copy_service.cc:433] P 4092902b3e674857a0147f1b4f2cdb86: ending tablet copy session 68a3c2b17ed94e06aba66ed4178132ab-2dc3b47b4ab145ccb9efbf1b5740c7f6 on tablet 2dc3b47b4ab145ccb9efbf1b5740c7f6 with peer 68a3c2b17ed94e06aba66ed4178132ab
I20240502 14:16:41.559852 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:41.583937 26406 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:41.584614 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:41.589814 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:41.619516 26687 raft_consensus.cc:1203] T a262ac98859b4208b70573662cb898cb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:16:41.645885 26556 raft_consensus.cc:1203] T 65b7c6aa6aac45079adcff73f465acdb P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:16:41.649464 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:41.659147 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:41.687533 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:41.701493 26687 raft_consensus.cc:1203] T dc3740d0671d42bc90eb9baeb02f5b0a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:16:41.888362 29440 raft_consensus.cc:1203] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:16:42.176095 29155 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:42.189059 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:42.232810 26865 maintenance_manager.cc:419] P e4cfb8fc5a474177af6848a6efa13a27: Scheduling FlushMRSOp(ce03ee354c54457487ea939301ccd90e): perf score=0.033345
I20240502 14:16:42.240794 26604 maintenance_manager.cc:419] P ba187e38a2754d60a12de9ac4856430a: Scheduling FlushMRSOp(058eec4efd564c348684857372c60127): perf score=0.033351
I20240502 14:16:42.272581 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:42.290356 26735 maintenance_manager.cc:419] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Scheduling FlushMRSOp(ce03ee354c54457487ea939301ccd90e): perf score=0.033364
I20240502 14:16:42.332489 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:42.337324 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:42.411770 26473 maintenance_manager.cc:419] P 89af7f423cf44717a08d6b142f0ed49d: Scheduling FlushMRSOp(058eec4efd564c348684857372c60127): perf score=0.033392
I20240502 14:16:42.436543 26406 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:42.485015 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:42.490969 26756 maintenance_manager.cc:643] P e4cfb8fc5a474177af6848a6efa13a27: FlushMRSOp(ce03ee354c54457487ea939301ccd90e) complete. Timing: real 0.258s user 0.048s sys 0.014s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":45484,"dirs.run_cpu_time_us":238,"dirs.run_wall_time_us":5362,"drs_written":1,"lbm_read_time_us":142,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1208,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":94563,"threads_started":2,"wal-append.queue_time_us":52280}
I20240502 14:16:42.493894 26865 maintenance_manager.cc:419] P e4cfb8fc5a474177af6848a6efa13a27: Scheduling FlushMRSOp(40aa23be0caa427e89e8fdaf905dc052): perf score=0.033413
I20240502 14:16:42.501543 26626 maintenance_manager.cc:643] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: FlushMRSOp(ce03ee354c54457487ea939301ccd90e) complete. Timing: real 0.211s user 0.034s sys 0.027s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":7218,"dirs.run_cpu_time_us":280,"dirs.run_wall_time_us":8885,"drs_written":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1013,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":47175,"threads_started":2,"wal-append.queue_time_us":33821}
I20240502 14:16:42.510124 26735 maintenance_manager.cc:419] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Scheduling FlushMRSOp(40aa23be0caa427e89e8fdaf905dc052): perf score=0.033418
I20240502 14:16:42.527686 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:42.564368 26495 maintenance_manager.cc:643] P ba187e38a2754d60a12de9ac4856430a: FlushMRSOp(058eec4efd564c348684857372c60127) complete. Timing: real 0.323s user 0.055s sys 0.014s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":79716,"dirs.run_cpu_time_us":236,"dirs.run_wall_time_us":5378,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1069,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":98943,"threads_started":2,"wal-append.queue_time_us":11725}
I20240502 14:16:42.568264 26604 maintenance_manager.cc:419] P ba187e38a2754d60a12de9ac4856430a: Scheduling UndoDeltaBlockGCOp(058eec4efd564c348684857372c60127): 4840 bytes on disk
I20240502 14:16:42.569643 26495 maintenance_manager.cc:643] P ba187e38a2754d60a12de9ac4856430a: UndoDeltaBlockGCOp(058eec4efd564c348684857372c60127) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240502 14:16:42.612298 26364 maintenance_manager.cc:643] P 89af7f423cf44717a08d6b142f0ed49d: FlushMRSOp(058eec4efd564c348684857372c60127) complete. Timing: real 0.200s user 0.034s sys 0.028s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":7695,"dirs.run_cpu_time_us":253,"dirs.run_wall_time_us":8026,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":852,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":14348,"threads_started":2,"wal-append.queue_time_us":6367}
I20240502 14:16:42.615536 26473 maintenance_manager.cc:419] P 89af7f423cf44717a08d6b142f0ed49d: Scheduling UndoDeltaBlockGCOp(058eec4efd564c348684857372c60127): 4840 bytes on disk
I20240502 14:16:42.616744 26364 maintenance_manager.cc:643] P 89af7f423cf44717a08d6b142f0ed49d: UndoDeltaBlockGCOp(058eec4efd564c348684857372c60127) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":167,"lbm_reads_lt_1ms":4}
I20240502 14:16:42.675666 26626 maintenance_manager.cc:643] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: FlushMRSOp(40aa23be0caa427e89e8fdaf905dc052) complete. Timing: real 0.156s user 0.046s sys 0.018s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":48,"dirs.run_cpu_time_us":210,"dirs.run_wall_time_us":1714,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":917,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":10505,"threads_started":1,"wal-append.queue_time_us":10606}
I20240502 14:16:42.678489 26735 maintenance_manager.cc:419] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Scheduling UndoDeltaBlockGCOp(ce03ee354c54457487ea939301ccd90e): 4925 bytes on disk
I20240502 14:16:42.679143 26626 maintenance_manager.cc:643] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: UndoDeltaBlockGCOp(ce03ee354c54457487ea939301ccd90e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20240502 14:16:42.681363 26735 maintenance_manager.cc:419] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Scheduling UndoDeltaBlockGCOp(40aa23be0caa427e89e8fdaf905dc052): 4586 bytes on disk
I20240502 14:16:42.683065 26626 maintenance_manager.cc:643] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: UndoDeltaBlockGCOp(40aa23be0caa427e89e8fdaf905dc052) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
I20240502 14:16:42.726981 26756 maintenance_manager.cc:643] P e4cfb8fc5a474177af6848a6efa13a27: FlushMRSOp(40aa23be0caa427e89e8fdaf905dc052) complete. Timing: real 0.233s user 0.032s sys 0.016s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":10224,"dirs.run_cpu_time_us":271,"dirs.run_wall_time_us":1960,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":910,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":7763,"threads_started":1,"wal-append.queue_time_us":7644}
I20240502 14:16:42.730619 26865 maintenance_manager.cc:419] P e4cfb8fc5a474177af6848a6efa13a27: Scheduling UndoDeltaBlockGCOp(ce03ee354c54457487ea939301ccd90e): 4925 bytes on disk
I20240502 14:16:42.731428 26756 maintenance_manager.cc:643] P e4cfb8fc5a474177af6848a6efa13a27: UndoDeltaBlockGCOp(ce03ee354c54457487ea939301ccd90e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240502 14:16:42.734874 26865 maintenance_manager.cc:419] P e4cfb8fc5a474177af6848a6efa13a27: Scheduling UndoDeltaBlockGCOp(40aa23be0caa427e89e8fdaf905dc052): 4586 bytes on disk
I20240502 14:16:42.737433 26756 maintenance_manager.cc:643] P e4cfb8fc5a474177af6848a6efa13a27: UndoDeltaBlockGCOp(40aa23be0caa427e89e8fdaf905dc052) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":1049,"lbm_reads_lt_1ms":4}
I20240502 14:16:42.822794 26815 consensus_queue.cc:237] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:16:42.825982 26687 raft_consensus.cc:1261] T dc3740d0671d42bc90eb9baeb02f5b0a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:42.827082 10501 consensus_queue.cc:1035] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:42.827250 29574 raft_consensus.cc:1261] T dc3740d0671d42bc90eb9baeb02f5b0a P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:42.827981 10501 consensus_queue.cc:1035] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:42.831655 10507 raft_consensus.cc:2932] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:42.832494 26687 raft_consensus.cc:2932] T dc3740d0671d42bc90eb9baeb02f5b0a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:42.841472 29575 raft_consensus.cc:2932] T dc3740d0671d42bc90eb9baeb02f5b0a P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:42.842669 26285 catalog_manager.cc:5484] T dc3740d0671d42bc90eb9baeb02f5b0a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:42.865826 29421 tablet_service.cc:1514] Processing DeleteTablet for tablet dc3740d0671d42bc90eb9baeb02f5b0a with delete_type TABLET_DATA_TOMBSTONED (TS 68a3c2b17ed94e06aba66ed4178132ab not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:40652
I20240502 14:16:42.867278 10733 tablet_replica.cc:330] T dc3740d0671d42bc90eb9baeb02f5b0a P 68a3c2b17ed94e06aba66ed4178132ab: stopping tablet replica
I20240502 14:16:42.867596 10733 raft_consensus.cc:2229] T dc3740d0671d42bc90eb9baeb02f5b0a P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:42.867825 10733 raft_consensus.cc:2258] T dc3740d0671d42bc90eb9baeb02f5b0a P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:42.868670 10733 ts_tablet_manager.cc:1875] T dc3740d0671d42bc90eb9baeb02f5b0a P 68a3c2b17ed94e06aba66ed4178132ab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:42.876432 10733 ts_tablet_manager.cc:1888] T dc3740d0671d42bc90eb9baeb02f5b0a P 68a3c2b17ed94e06aba66ed4178132ab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:16:42.876564 10733 log.cc:1196] T dc3740d0671d42bc90eb9baeb02f5b0a P 68a3c2b17ed94e06aba66ed4178132ab: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal/wals/dc3740d0671d42bc90eb9baeb02f5b0a
I20240502 14:16:42.877413 26272 catalog_manager.cc:4830] TS 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): tablet dc3740d0671d42bc90eb9baeb02f5b0a (table rebalancer_extra_table_6 [id=fe880af4989f416d82ae711576173990]) successfully deleted
I20240502 14:16:43.222695 29155 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:43.223860 26406 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:43.229130 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:43.321882 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:43.349018 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:43.364477 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:43.365291 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:16:43.397078 29511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.196:32843, user_credentials={real_user=slave}} blocked reactor thread for 67106.5us
I20240502 14:16:43.403072 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240502 14:16:43.426772 29245 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.199:41781, user_credentials={real_user=slave}} blocked reactor thread for 60893.5us
I20240502 14:16:43.635525 29575 consensus_queue.cc:237] T a262ac98859b4208b70573662cb898cb P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:16:43.642475 26687 raft_consensus.cc:1261] T a262ac98859b4208b70573662cb898cb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:43.642475 29174 raft_consensus.cc:1261] T a262ac98859b4208b70573662cb898cb P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:43.643052 10400 consensus_queue.cc:1035] T a262ac98859b4208b70573662cb898cb P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:43.643458 10789 consensus_queue.cc:1035] T a262ac98859b4208b70573662cb898cb P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:43.654655 10789 raft_consensus.cc:2932] T a262ac98859b4208b70573662cb898cb P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:43.658412 26687 raft_consensus.cc:2932] T a262ac98859b4208b70573662cb898cb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:43.661595 26283 catalog_manager.cc:5484] T a262ac98859b4208b70573662cb898cb P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:43.661749 29174 raft_consensus.cc:2932] T a262ac98859b4208b70573662cb898cb P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:43.696471 29421 tablet_service.cc:1514] Processing DeleteTablet for tablet a262ac98859b4208b70573662cb898cb with delete_type TABLET_DATA_TOMBSTONED (TS 68a3c2b17ed94e06aba66ed4178132ab not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:40652
I20240502 14:16:43.713783 10824 tablet_replica.cc:330] T a262ac98859b4208b70573662cb898cb P 68a3c2b17ed94e06aba66ed4178132ab: stopping tablet replica
I20240502 14:16:43.714046 10824 raft_consensus.cc:2229] T a262ac98859b4208b70573662cb898cb P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:43.714449 10824 raft_consensus.cc:2258] T a262ac98859b4208b70573662cb898cb P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:43.716567 10824 ts_tablet_manager.cc:1875] T a262ac98859b4208b70573662cb898cb P 68a3c2b17ed94e06aba66ed4178132ab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:43.724769 10824 ts_tablet_manager.cc:1888] T a262ac98859b4208b70573662cb898cb P 68a3c2b17ed94e06aba66ed4178132ab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:16:43.725005 10824 log.cc:1196] T a262ac98859b4208b70573662cb898cb P 68a3c2b17ed94e06aba66ed4178132ab: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal/wals/a262ac98859b4208b70573662cb898cb
I20240502 14:16:43.725860 26272 catalog_manager.cc:4830] TS 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): tablet a262ac98859b4208b70573662cb898cb (table rebalancer_extra_table_16 [id=15e0d2ef8fa244be97447fb977d23693]) successfully deleted
I20240502 14:16:44.045198 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:44.051318 29155 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:44.096648 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240502 14:16:44.165089 29511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 54743.7us
I20240502 14:16:44.193464 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:44.273156 26406 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:44.345254 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:44.374835 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:44.409073 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:44.568004 29174 consensus_queue.cc:237] T 65b7c6aa6aac45079adcff73f465acdb P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:16:44.570392 26556 raft_consensus.cc:1261] T 65b7c6aa6aac45079adcff73f465acdb P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:44.570896 26686 raft_consensus.cc:1261] T 65b7c6aa6aac45079adcff73f465acdb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:44.571630 10677 consensus_queue.cc:1035] T 65b7c6aa6aac45079adcff73f465acdb P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:44.572222 10105 consensus_queue.cc:1035] T 65b7c6aa6aac45079adcff73f465acdb P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:44.576079 10105 raft_consensus.cc:2932] T 65b7c6aa6aac45079adcff73f465acdb P 1cb67158133e4648a6eceba32d4dbba3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:16:44.576706 26686 raft_consensus.cc:2932] T 65b7c6aa6aac45079adcff73f465acdb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:16:44.577795 26556 raft_consensus.cc:2932] T 65b7c6aa6aac45079adcff73f465acdb P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:16:44.584143 26277 catalog_manager.cc:5484] T 65b7c6aa6aac45079adcff73f465acdb P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New cstate: current_term: 1 leader_uuid: "1cb67158133e4648a6eceba32d4dbba3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:16:44.599648 29287 tablet_service.cc:1514] Processing DeleteTablet for tablet 65b7c6aa6aac45079adcff73f465acdb with delete_type TABLET_DATA_TOMBSTONED (TS c1100e16b3b9495ca61d4f95e89b6a89 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:59160
I20240502 14:16:44.600813 10916 tablet_replica.cc:330] T 65b7c6aa6aac45079adcff73f465acdb P c1100e16b3b9495ca61d4f95e89b6a89: stopping tablet replica
I20240502 14:16:44.601125 10916 raft_consensus.cc:2229] T 65b7c6aa6aac45079adcff73f465acdb P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:44.601414 10916 raft_consensus.cc:2258] T 65b7c6aa6aac45079adcff73f465acdb P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:44.602170 10916 ts_tablet_manager.cc:1875] T 65b7c6aa6aac45079adcff73f465acdb P c1100e16b3b9495ca61d4f95e89b6a89: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:44.609534 10916 ts_tablet_manager.cc:1888] T 65b7c6aa6aac45079adcff73f465acdb P c1100e16b3b9495ca61d4f95e89b6a89: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:16:44.609705 10916 log.cc:1196] T 65b7c6aa6aac45079adcff73f465acdb P c1100e16b3b9495ca61d4f95e89b6a89: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal/wals/65b7c6aa6aac45079adcff73f465acdb
I20240502 14:16:44.611372 26271 catalog_manager.cc:4830] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): tablet 65b7c6aa6aac45079adcff73f465acdb (table rebalancer_extra_table_36 [id=123f30b9ad634d07ada549c46031d008]) successfully deleted
I20240502 14:16:44.648536 26473 maintenance_manager.cc:419] P 89af7f423cf44717a08d6b142f0ed49d: Scheduling FlushMRSOp(bb3e4ec75888419aa4e065cf5513ae5b): perf score=0.033350
I20240502 14:16:44.697325 26364 maintenance_manager.cc:643] P 89af7f423cf44717a08d6b142f0ed49d: FlushMRSOp(bb3e4ec75888419aa4e065cf5513ae5b) complete. Timing: real 0.048s user 0.036s sys 0.005s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":6553,"dirs.run_cpu_time_us":255,"dirs.run_wall_time_us":10105,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1287,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":7672,"threads_started":2,"wal-append.queue_time_us":871}
I20240502 14:16:44.702244 26473 maintenance_manager.cc:419] P 89af7f423cf44717a08d6b142f0ed49d: Scheduling FlushMRSOp(981e25d29abc44299dc115d6df835c58): perf score=0.033358
I20240502 14:16:44.707721 26735 maintenance_manager.cc:419] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Scheduling FlushMRSOp(cf52d0f868fa45418af48639e12f3282): perf score=0.033352
I20240502 14:16:44.764062 26364 maintenance_manager.cc:643] P 89af7f423cf44717a08d6b142f0ed49d: FlushMRSOp(981e25d29abc44299dc115d6df835c58) complete. Timing: real 0.061s user 0.038s sys 0.013s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":55,"dirs.run_cpu_time_us":199,"dirs.run_wall_time_us":1721,"drs_written":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4,"lbm_write_time_us":902,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":681,"threads_started":1,"wal-append.queue_time_us":854}
I20240502 14:16:44.767266 26865 maintenance_manager.cc:419] P e4cfb8fc5a474177af6848a6efa13a27: Scheduling FlushMRSOp(bb3e4ec75888419aa4e065cf5513ae5b): perf score=0.033368
I20240502 14:16:44.767482 26473 maintenance_manager.cc:419] P 89af7f423cf44717a08d6b142f0ed49d: Scheduling UndoDeltaBlockGCOp(bb3e4ec75888419aa4e065cf5513ae5b): 4925 bytes on disk
I20240502 14:16:44.768882 26364 maintenance_manager.cc:643] P 89af7f423cf44717a08d6b142f0ed49d: UndoDeltaBlockGCOp(bb3e4ec75888419aa4e065cf5513ae5b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4}
I20240502 14:16:44.774443 26473 maintenance_manager.cc:419] P 89af7f423cf44717a08d6b142f0ed49d: Scheduling UndoDeltaBlockGCOp(981e25d29abc44299dc115d6df835c58): 4840 bytes on disk
I20240502 14:16:44.776486 26364 maintenance_manager.cc:643] P 89af7f423cf44717a08d6b142f0ed49d: UndoDeltaBlockGCOp(981e25d29abc44299dc115d6df835c58) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":347,"lbm_reads_lt_1ms":4}
I20240502 14:16:44.811735 26626 maintenance_manager.cc:643] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: FlushMRSOp(cf52d0f868fa45418af48639e12f3282) complete. Timing: real 0.103s user 0.051s sys 0.005s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":6726,"dirs.run_cpu_time_us":316,"dirs.run_wall_time_us":11925,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":875,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":44758,"threads_started":2,"wal-append.queue_time_us":37692}
I20240502 14:16:44.814918 26735 maintenance_manager.cc:419] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Scheduling UndoDeltaBlockGCOp(cf52d0f868fa45418af48639e12f3282): 4586 bytes on disk
I20240502 14:16:44.815732 26626 maintenance_manager.cc:643] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: UndoDeltaBlockGCOp(cf52d0f868fa45418af48639e12f3282) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":118,"lbm_reads_lt_1ms":4}
I20240502 14:16:44.853121 26604 maintenance_manager.cc:419] P ba187e38a2754d60a12de9ac4856430a: Scheduling FlushMRSOp(cf52d0f868fa45418af48639e12f3282): perf score=0.033392
I20240502 14:16:44.919167 26756 maintenance_manager.cc:643] P e4cfb8fc5a474177af6848a6efa13a27: FlushMRSOp(bb3e4ec75888419aa4e065cf5513ae5b) complete. Timing: real 0.152s user 0.066s sys 0.043s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":68170,"dirs.run_cpu_time_us":227,"dirs.run_wall_time_us":2175,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1152,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":77676,"threads_started":2,"wal-append.queue_time_us":9711}
I20240502 14:16:44.922104 26865 maintenance_manager.cc:419] P e4cfb8fc5a474177af6848a6efa13a27: Scheduling UndoDeltaBlockGCOp(bb3e4ec75888419aa4e065cf5513ae5b): 4925 bytes on disk
I20240502 14:16:44.923033 26756 maintenance_manager.cc:643] P e4cfb8fc5a474177af6848a6efa13a27: UndoDeltaBlockGCOp(bb3e4ec75888419aa4e065cf5513ae5b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240502 14:16:44.926256 26865 maintenance_manager.cc:419] P e4cfb8fc5a474177af6848a6efa13a27: Scheduling FlushMRSOp(981e25d29abc44299dc115d6df835c58): perf score=0.033409
I20240502 14:16:44.934079 26495 maintenance_manager.cc:643] P ba187e38a2754d60a12de9ac4856430a: FlushMRSOp(cf52d0f868fa45418af48639e12f3282) complete. Timing: real 0.081s user 0.041s sys 0.002s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":5773,"dirs.run_cpu_time_us":269,"dirs.run_wall_time_us":5619,"drs_written":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4,"lbm_write_time_us":884,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":6931,"threads_started":2,"wal-append.queue_time_us":596}
I20240502 14:16:44.939198 26604 maintenance_manager.cc:419] P ba187e38a2754d60a12de9ac4856430a: Scheduling UndoDeltaBlockGCOp(cf52d0f868fa45418af48639e12f3282): 4586 bytes on disk
I20240502 14:16:44.940237 26495 maintenance_manager.cc:643] P ba187e38a2754d60a12de9ac4856430a: UndoDeltaBlockGCOp(cf52d0f868fa45418af48639e12f3282) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":165,"lbm_reads_lt_1ms":4}
I20240502 14:16:44.982463 26756 maintenance_manager.cc:643] P e4cfb8fc5a474177af6848a6efa13a27: FlushMRSOp(981e25d29abc44299dc115d6df835c58) complete. Timing: real 0.056s user 0.044s sys 0.003s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":53,"dirs.run_cpu_time_us":218,"dirs.run_wall_time_us":1643,"drs_written":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1160,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":7222,"threads_started":1,"wal-append.queue_time_us":7086}
I20240502 14:16:44.986625 26865 maintenance_manager.cc:419] P e4cfb8fc5a474177af6848a6efa13a27: Scheduling UndoDeltaBlockGCOp(981e25d29abc44299dc115d6df835c58): 4840 bytes on disk
I20240502 14:16:45.015189 26756 maintenance_manager.cc:643] P e4cfb8fc5a474177af6848a6efa13a27: UndoDeltaBlockGCOp(981e25d29abc44299dc115d6df835c58) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":189,"lbm_reads_lt_1ms":4}
I20240502 14:16:45.230024 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 14:16:45.258040 26363 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.196:32843, user_credentials={real_user=slave}} blocked reactor thread for 66135.1us
I20240502 14:16:45.294319 29155 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:45.308492 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:45.323678 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:45.324705 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:45.340972 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:45.412441 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:45.434163 26406 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:45.655201 26427 consensus_queue.cc:237] T a04b8c49626e491b845d9c81142ac659 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:16:45.669800 26686 raft_consensus.cc:1261] T a04b8c49626e491b845d9c81142ac659 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:45.669952 26816 raft_consensus.cc:1261] T a04b8c49626e491b845d9c81142ac659 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:45.670862 9440 consensus_queue.cc:1035] T a04b8c49626e491b845d9c81142ac659 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:45.671236 10844 consensus_queue.cc:1035] T a04b8c49626e491b845d9c81142ac659 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:45.674858 9440 raft_consensus.cc:2932] T a04b8c49626e491b845d9c81142ac659 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:45.675554 26687 raft_consensus.cc:2932] T a04b8c49626e491b845d9c81142ac659 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:45.676719 26816 raft_consensus.cc:2932] T a04b8c49626e491b845d9c81142ac659 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:45.684345 26285 catalog_manager.cc:5484] T a04b8c49626e491b845d9c81142ac659 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:45.712286 29421 tablet_service.cc:1514] Processing DeleteTablet for tablet a04b8c49626e491b845d9c81142ac659 with delete_type TABLET_DATA_TOMBSTONED (TS 68a3c2b17ed94e06aba66ed4178132ab not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:40652
I20240502 14:16:45.715669 11026 tablet_replica.cc:330] T a04b8c49626e491b845d9c81142ac659 P 68a3c2b17ed94e06aba66ed4178132ab: stopping tablet replica
I20240502 14:16:45.716542 11026 raft_consensus.cc:2229] T a04b8c49626e491b845d9c81142ac659 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:45.716903 11026 raft_consensus.cc:2258] T a04b8c49626e491b845d9c81142ac659 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:45.718159 11026 ts_tablet_manager.cc:1875] T a04b8c49626e491b845d9c81142ac659 P 68a3c2b17ed94e06aba66ed4178132ab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:45.733938 11026 ts_tablet_manager.cc:1888] T a04b8c49626e491b845d9c81142ac659 P 68a3c2b17ed94e06aba66ed4178132ab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:16:45.734143 11026 log.cc:1196] T a04b8c49626e491b845d9c81142ac659 P 68a3c2b17ed94e06aba66ed4178132ab: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal/wals/a04b8c49626e491b845d9c81142ac659
I20240502 14:16:45.735157 26272 catalog_manager.cc:4830] TS 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): tablet a04b8c49626e491b845d9c81142ac659 (table rebalancer_extra_table_28 [id=0dc6f24955544368ae080f1a95f9589b]) successfully deleted
I20240502 14:16:46.109427 29155 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:46.140169 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:46.143621 26406 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:46.223357 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:16:46.273983 29378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.194:36025, user_credentials={real_user=slave}} blocked reactor thread for 87767.9us
I20240502 14:16:46.277985 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:46.287068 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:46.297430 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 14:16:46.339314 29378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.198:43659, user_credentials={real_user=slave}} blocked reactor thread for 48296.1us
I20240502 14:16:46.353360 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 14:16:46.369797 26625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.196:32843, user_credentials={real_user=slave}} blocked reactor thread for 187066us
I20240502 14:16:46.584733 29443 consensus_queue.cc:237] T eb065120d1304a2a81ce932934b911f3 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:46.586879 26425 raft_consensus.cc:1261] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:46.587044 29175 raft_consensus.cc:1261] T eb065120d1304a2a81ce932934b911f3 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:46.587464 11085 consensus_queue.cc:1035] T eb065120d1304a2a81ce932934b911f3 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:46.587765 11084 consensus_queue.cc:1035] T eb065120d1304a2a81ce932934b911f3 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:46.591514 11085 raft_consensus.cc:2932] T eb065120d1304a2a81ce932934b911f3 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:46.592381 26425 raft_consensus.cc:2932] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:46.592782 29175 raft_consensus.cc:2932] T eb065120d1304a2a81ce932934b911f3 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:46.599267 26284 catalog_manager.cc:5484] T eb065120d1304a2a81ce932934b911f3 P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New cstate: current_term: 1 leader_uuid: "68a3c2b17ed94e06aba66ed4178132ab" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:46.633145 29287 tablet_service.cc:1514] Processing DeleteTablet for tablet eb065120d1304a2a81ce932934b911f3 with delete_type TABLET_DATA_TOMBSTONED (TS c1100e16b3b9495ca61d4f95e89b6a89 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:59160
I20240502 14:16:46.633850 11116 tablet_replica.cc:330] T eb065120d1304a2a81ce932934b911f3 P c1100e16b3b9495ca61d4f95e89b6a89: stopping tablet replica
I20240502 14:16:46.634093 11116 raft_consensus.cc:2229] T eb065120d1304a2a81ce932934b911f3 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:46.634712 11116 raft_consensus.cc:2258] T eb065120d1304a2a81ce932934b911f3 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:46.636005 11116 ts_tablet_manager.cc:1875] T eb065120d1304a2a81ce932934b911f3 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:46.646754 11116 ts_tablet_manager.cc:1888] T eb065120d1304a2a81ce932934b911f3 P c1100e16b3b9495ca61d4f95e89b6a89: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:16:46.647010 11116 log.cc:1196] T eb065120d1304a2a81ce932934b911f3 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal/wals/eb065120d1304a2a81ce932934b911f3
I20240502 14:16:46.649551 26271 catalog_manager.cc:4830] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): tablet eb065120d1304a2a81ce932934b911f3 (table rebalancer_extra_table_18 [id=800c8a30773545b3a7670200a5873c0e]) successfully deleted
I20240502 14:16:46.979233 29155 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:47.036083 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:47.037196 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:47.087553 26406 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 14:16:47.103870 26624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.197:38715, user_credentials={real_user=slave}} blocked reactor thread for 80039.4us
I20240502 14:16:47.118162 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:47.126705 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:47.130118 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:47.221894 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:47.376682 29574 consensus_queue.cc:237] T fc6401f0ab8c485399325df7c8dd96ad P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:16:47.378765 26687 raft_consensus.cc:1261] T fc6401f0ab8c485399325df7c8dd96ad P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:47.378942 26556 raft_consensus.cc:1261] T fc6401f0ab8c485399325df7c8dd96ad P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:47.379439 10789 consensus_queue.cc:1035] T fc6401f0ab8c485399325df7c8dd96ad P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:47.379719 11158 consensus_queue.cc:1035] T fc6401f0ab8c485399325df7c8dd96ad P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:47.398054 11158 raft_consensus.cc:2932] T fc6401f0ab8c485399325df7c8dd96ad P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:47.399699 26687 raft_consensus.cc:2932] T fc6401f0ab8c485399325df7c8dd96ad P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:47.403138 26556 raft_consensus.cc:2932] T fc6401f0ab8c485399325df7c8dd96ad P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:47.406469 26285 catalog_manager.cc:5484] T fc6401f0ab8c485399325df7c8dd96ad P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:47.441293 29421 tablet_service.cc:1514] Processing DeleteTablet for tablet fc6401f0ab8c485399325df7c8dd96ad with delete_type TABLET_DATA_TOMBSTONED (TS 68a3c2b17ed94e06aba66ed4178132ab not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:40652
I20240502 14:16:47.442798 11209 tablet_replica.cc:330] T fc6401f0ab8c485399325df7c8dd96ad P 68a3c2b17ed94e06aba66ed4178132ab: stopping tablet replica
I20240502 14:16:47.443007 11209 raft_consensus.cc:2229] T fc6401f0ab8c485399325df7c8dd96ad P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:47.443228 11209 raft_consensus.cc:2258] T fc6401f0ab8c485399325df7c8dd96ad P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:47.443882 11209 ts_tablet_manager.cc:1875] T fc6401f0ab8c485399325df7c8dd96ad P 68a3c2b17ed94e06aba66ed4178132ab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:47.451320 11209 ts_tablet_manager.cc:1888] T fc6401f0ab8c485399325df7c8dd96ad P 68a3c2b17ed94e06aba66ed4178132ab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:16:47.451439 11209 log.cc:1196] T fc6401f0ab8c485399325df7c8dd96ad P 68a3c2b17ed94e06aba66ed4178132ab: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal/wals/fc6401f0ab8c485399325df7c8dd96ad
I20240502 14:16:47.452211 26272 catalog_manager.cc:4830] TS 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): tablet fc6401f0ab8c485399325df7c8dd96ad (table rebalancer_extra_table_20 [id=2ccf4597fbc1491f87f24cd5733a46d2]) successfully deleted
I20240502 14:16:47.807826 29155 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:47.831236 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:47.873474 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:47.876619 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:47.887049 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:47.905098 26406 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:47.922084 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:47.969393 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:48.137332 26425 consensus_queue.cc:237] T d344462241604470a48a443b396177a7 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:16:48.139587 29174 raft_consensus.cc:1261] T d344462241604470a48a443b396177a7 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:48.139587 26556 raft_consensus.cc:1261] T d344462241604470a48a443b396177a7 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:48.140177 11181 consensus_queue.cc:1035] T d344462241604470a48a443b396177a7 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:48.140506 10989 consensus_queue.cc:1035] T d344462241604470a48a443b396177a7 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:48.144425 11181 raft_consensus.cc:2932] T d344462241604470a48a443b396177a7 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:16:48.145644 26556 raft_consensus.cc:2932] T d344462241604470a48a443b396177a7 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:16:48.152875 26283 catalog_manager.cc:5484] T d344462241604470a48a443b396177a7 P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:16:48.177860 29174 raft_consensus.cc:2932] T d344462241604470a48a443b396177a7 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:16:48.198776 29287 tablet_service.cc:1514] Processing DeleteTablet for tablet d344462241604470a48a443b396177a7 with delete_type TABLET_DATA_TOMBSTONED (TS c1100e16b3b9495ca61d4f95e89b6a89 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:59160
I20240502 14:16:48.210714 11295 tablet_replica.cc:330] T d344462241604470a48a443b396177a7 P c1100e16b3b9495ca61d4f95e89b6a89: stopping tablet replica
I20240502 14:16:48.211042 11295 raft_consensus.cc:2229] T d344462241604470a48a443b396177a7 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:48.211652 11295 raft_consensus.cc:2258] T d344462241604470a48a443b396177a7 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:48.213214 11295 ts_tablet_manager.cc:1875] T d344462241604470a48a443b396177a7 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:48.225109 11295 ts_tablet_manager.cc:1888] T d344462241604470a48a443b396177a7 P c1100e16b3b9495ca61d4f95e89b6a89: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:16:48.231204 11295 log.cc:1196] T d344462241604470a48a443b396177a7 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal/wals/d344462241604470a48a443b396177a7
I20240502 14:16:48.233569 26271 catalog_manager.cc:4830] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): tablet d344462241604470a48a443b396177a7 (table rebalancer_extra_table_22 [id=10bc76506bf848969a3acfd452e07b7f]) successfully deleted
I20240502 14:16:48.613571 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:48.673862 29155 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:48.676785 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:48.676973 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:48.705468 26406 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:48.797798 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:48.818044 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:48.936478 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 14:16:48.956887 29245 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.200:46097, user_credentials={real_user=slave}} blocked reactor thread for 65652.1us
I20240502 14:16:49.088356 29574 consensus_queue.cc:237] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:16:49.091115 29443 raft_consensus.cc:1261] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:49.091657 26425 raft_consensus.cc:1261] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:49.091634 11158 consensus_queue.cc:1035] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:49.092201 11335 consensus_queue.cc:1035] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:49.095371 11335 raft_consensus.cc:2932] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:49.099871 29443 raft_consensus.cc:2932] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:49.103307 26282 catalog_manager.cc:5484] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:49.111030 26425 raft_consensus.cc:2932] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:49.135205 29287 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dc3b47b4ab145ccb9efbf1b5740c7f6 with delete_type TABLET_DATA_TOMBSTONED (TS c1100e16b3b9495ca61d4f95e89b6a89 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:59160
I20240502 14:16:49.136137 11394 tablet_replica.cc:330] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P c1100e16b3b9495ca61d4f95e89b6a89: stopping tablet replica
I20240502 14:16:49.137725 11394 raft_consensus.cc:2229] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:49.138077 11394 raft_consensus.cc:2258] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:49.138892 11394 ts_tablet_manager.cc:1875] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:49.146091 11394 ts_tablet_manager.cc:1888] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P c1100e16b3b9495ca61d4f95e89b6a89: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:16:49.146308 11394 log.cc:1196] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-5/wal/wals/2dc3b47b4ab145ccb9efbf1b5740c7f6
I20240502 14:16:49.149539 26271 catalog_manager.cc:4830] TS c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): tablet 2dc3b47b4ab145ccb9efbf1b5740c7f6 (table rebalancer_extra_table_48 [id=53d47c43f2f14c30b29a5147461aafae]) successfully deleted
I20240502 14:16:49.315138 29574 consensus_queue.cc:237] T 88b726d7142846d6893f4f3b763c0565 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:16:49.318243 26816 raft_consensus.cc:1261] T 88b726d7142846d6893f4f3b763c0565 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:49.318828 29443 raft_consensus.cc:1261] T 88b726d7142846d6893f4f3b763c0565 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:49.319587 11335 consensus_queue.cc:1035] T 88b726d7142846d6893f4f3b763c0565 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:49.319881 11159 consensus_queue.cc:1035] T 88b726d7142846d6893f4f3b763c0565 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:16:49.320490 29511 consensus_peers.cc:485] T 88b726d7142846d6893f4f3b763c0565 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): Couldn't send request to peer 69ab78b2a1b74d2bbf2cdeafa0a4af23. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 88b726d7142846d6893f4f3b763c0565. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:49.324476 11335 raft_consensus.cc:2932] T 88b726d7142846d6893f4f3b763c0565 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:49.325147 29443 raft_consensus.cc:2932] T 88b726d7142846d6893f4f3b763c0565 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:49.329361 26816 raft_consensus.cc:2932] T 88b726d7142846d6893f4f3b763c0565 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:49.332532 26280 catalog_manager.cc:5484] T 88b726d7142846d6893f4f3b763c0565 P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:49.697278 29155 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:49.760095 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:49.769788 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:49.771586 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:49.826476 11507 ts_tablet_manager.cc:902] T 88b726d7142846d6893f4f3b763c0565 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Initiating tablet copy from peer 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:16:49.869251 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:49.890162 11507 tablet_copy_client.cc:308] T 88b726d7142846d6893f4f3b763c0565 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.200:46097
I20240502 14:16:49.890928 29594 tablet_copy_service.cc:140] P 4092902b3e674857a0147f1b4f2cdb86: Received BeginTabletCopySession request for tablet 88b726d7142846d6893f4f3b763c0565 from peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 ({username='slave'} at 127.19.94.195:39709)
I20240502 14:16:49.891094 29594 tablet_copy_service.cc:161] P 4092902b3e674857a0147f1b4f2cdb86: Beginning new tablet copy session on tablet 88b726d7142846d6893f4f3b763c0565 from peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 at {username='slave'} at 127.19.94.195:39709: session id = 69ab78b2a1b74d2bbf2cdeafa0a4af23-88b726d7142846d6893f4f3b763c0565
I20240502 14:16:49.893313 26406 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:49.894057 29594 tablet_copy_source_session.cc:205] T 88b726d7142846d6893f4f3b763c0565 P 4092902b3e674857a0147f1b4f2cdb86: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:49.895506 11507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88b726d7142846d6893f4f3b763c0565. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:49.904284 11507 tablet_copy_client.cc:783] T 88b726d7142846d6893f4f3b763c0565 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:49.904595 11507 tablet_copy_client.cc:647] T 88b726d7142846d6893f4f3b763c0565 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:49.906397 11507 tablet_copy_client.cc:523] T 88b726d7142846d6893f4f3b763c0565 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:49.910343 11507 tablet_bootstrap.cc:492] T 88b726d7142846d6893f4f3b763c0565 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:16:49.921554 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:49.921572 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:49.923662 11507 tablet_bootstrap.cc:492] T 88b726d7142846d6893f4f3b763c0565 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:49.924283 11507 tablet_bootstrap.cc:492] T 88b726d7142846d6893f4f3b763c0565 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap complete.
I20240502 14:16:49.924588 11507 ts_tablet_manager.cc:1367] T 88b726d7142846d6893f4f3b763c0565 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.003s
I20240502 14:16:49.925495 11507 raft_consensus.cc:348] T 88b726d7142846d6893f4f3b763c0565 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:16:49.925740 11507 raft_consensus.cc:729] T 88b726d7142846d6893f4f3b763c0565 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:16:49.926004 11507 consensus_queue.cc:260] T 88b726d7142846d6893f4f3b763c0565 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:16:49.928141 11507 ts_tablet_manager.cc:1398] T 88b726d7142846d6893f4f3b763c0565 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:16:49.929818 29594 tablet_copy_service.cc:342] P 4092902b3e674857a0147f1b4f2cdb86: Request end of tablet copy session 69ab78b2a1b74d2bbf2cdeafa0a4af23-88b726d7142846d6893f4f3b763c0565 received from {username='slave'} at 127.19.94.195:39709
I20240502 14:16:49.929939 29594 tablet_copy_service.cc:433] P 4092902b3e674857a0147f1b4f2cdb86: ending tablet copy session 69ab78b2a1b74d2bbf2cdeafa0a4af23-88b726d7142846d6893f4f3b763c0565 on tablet 88b726d7142846d6893f4f3b763c0565 with peer 69ab78b2a1b74d2bbf2cdeafa0a4af23
I20240502 14:16:50.324369 26686 raft_consensus.cc:1203] T 88b726d7142846d6893f4f3b763c0565 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:16:50.538000 29155 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:50.573056 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:50.579631 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:50.633224 26406 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:50.683120 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:50.695946 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:50.700246 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:50.730473 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:50.899713 29574 consensus_queue.cc:237] T 88b726d7142846d6893f4f3b763c0565 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:16:50.903352 26686 raft_consensus.cc:1261] T 88b726d7142846d6893f4f3b763c0565 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:50.903421 26815 raft_consensus.cc:1261] T 88b726d7142846d6893f4f3b763c0565 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:16:50.903941 11158 consensus_queue.cc:1035] T 88b726d7142846d6893f4f3b763c0565 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:50.904352 11335 consensus_queue.cc:1035] T 88b726d7142846d6893f4f3b763c0565 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:16:50.907007 11158 raft_consensus.cc:2932] T 88b726d7142846d6893f4f3b763c0565 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:50.907631 26815 raft_consensus.cc:2932] T 88b726d7142846d6893f4f3b763c0565 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:50.908756 26686 raft_consensus.cc:2932] T 88b726d7142846d6893f4f3b763c0565 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:50.915248 26281 catalog_manager.cc:5484] T 88b726d7142846d6893f4f3b763c0565 P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: config changed from index 2 to 3, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) evicted. New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:50.926226 29421 tablet_service.cc:1514] Processing DeleteTablet for tablet 88b726d7142846d6893f4f3b763c0565 with delete_type TABLET_DATA_TOMBSTONED (TS 68a3c2b17ed94e06aba66ed4178132ab not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:40652
I20240502 14:16:50.926980 11613 tablet_replica.cc:330] T 88b726d7142846d6893f4f3b763c0565 P 68a3c2b17ed94e06aba66ed4178132ab: stopping tablet replica
I20240502 14:16:50.927192 11613 raft_consensus.cc:2229] T 88b726d7142846d6893f4f3b763c0565 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:16:50.927428 11613 raft_consensus.cc:2258] T 88b726d7142846d6893f4f3b763c0565 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:16:50.928098 11613 ts_tablet_manager.cc:1875] T 88b726d7142846d6893f4f3b763c0565 P 68a3c2b17ed94e06aba66ed4178132ab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:16:50.933724 11613 ts_tablet_manager.cc:1888] T 88b726d7142846d6893f4f3b763c0565 P 68a3c2b17ed94e06aba66ed4178132ab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:16:50.933835 11613 log.cc:1196] T 88b726d7142846d6893f4f3b763c0565 P 68a3c2b17ed94e06aba66ed4178132ab: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-6/wal/wals/88b726d7142846d6893f4f3b763c0565
I20240502 14:16:50.934610 26272 catalog_manager.cc:4830] TS 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): tablet 88b726d7142846d6893f4f3b763c0565 (table rebalancer_extra_table_38 [id=7a3bd56fd0ef4a5f89e69991dd8d785a]) successfully deleted
I20240502 14:16:51.532756 29155 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:51.556504 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:51.585937 26406 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:51.603047 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:51.633839 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:51.658551 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:51.706218 29287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:51.783350 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:51.959789 26425 consensus_queue.cc:237] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:16:51.962677 29175 raft_consensus.cc:1261] T 0fc2e6c74dc34a45ae1ec46608642880 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:51.962711 26557 raft_consensus.cc:1261] T 0fc2e6c74dc34a45ae1ec46608642880 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 14:16:51.963542 26363 consensus_peers.cc:485] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097): Couldn't send request to peer 4092902b3e674857a0147f1b4f2cdb86. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0fc2e6c74dc34a45ae1ec46608642880. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:51.963721 11490 consensus_queue.cc:1035] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:51.964318 11490 consensus_queue.cc:1035] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:51.968968 11619 raft_consensus.cc:2932] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:16:51.969599 26556 raft_consensus.cc:2932] T 0fc2e6c74dc34a45ae1ec46608642880 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:16:51.977262 26285 catalog_manager.cc:5484] T 0fc2e6c74dc34a45ae1ec46608642880 P ba187e38a2754d60a12de9ac4856430a reported cstate change: config changed from index 5 to 6, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:16:51.979970 29175 raft_consensus.cc:2932] T 0fc2e6c74dc34a45ae1ec46608642880 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:16:52.026934 26556 consensus_queue.cc:237] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:16:52.031114 26815 raft_consensus.cc:1261] T 4bf6e7410afa4c048f5842ec2a426d77 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 14:16:52.032917 26493 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): Couldn't send request to peer 1cb67158133e4648a6eceba32d4dbba3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4bf6e7410afa4c048f5842ec2a426d77. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:52.034070 29438 raft_consensus.cc:1261] T 4bf6e7410afa4c048f5842ec2a426d77 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:52.034751 11717 consensus_queue.cc:1035] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:52.035308 9923 consensus_queue.cc:1035] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:52.045917 9923 raft_consensus.cc:2932] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:16:52.046772 29438 raft_consensus.cc:2932] T 4bf6e7410afa4c048f5842ec2a426d77 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:16:52.052034 26815 raft_consensus.cc:2932] T 4bf6e7410afa4c048f5842ec2a426d77 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:16:52.054752 26285 catalog_manager.cc:5484] T 4bf6e7410afa4c048f5842ec2a426d77 P 68a3c2b17ed94e06aba66ed4178132ab reported cstate change: config changed from index 5 to 6, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:16:52.133499 29438 consensus_queue.cc:237] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:16:52.139035 26425 raft_consensus.cc:1261] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:16:52.139096 29378 consensus_peers.cc:485] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:52.139940 26815 raft_consensus.cc:1261] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:52.140072 11458 consensus_queue.cc:1035] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:52.140590 11676 consensus_queue.cc:1035] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:52.143509 11458 raft_consensus.cc:2932] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:52.144222 26815 raft_consensus.cc:2932] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:52.149262 26425 raft_consensus.cc:2932] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:52.153796 26276 catalog_manager.cc:5484] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab reported cstate change: config changed from index 3 to 4, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New cstate: current_term: 1 leader_uuid: "68a3c2b17ed94e06aba66ed4178132ab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:52.227815 26686 consensus_queue.cc:237] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:52.230940 26815 raft_consensus.cc:1261] T 25b56a948668442da540ca7a198f9fbb P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:52.231228 29176 raft_consensus.cc:1261] T 25b56a948668442da540ca7a198f9fbb P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:52.231694 11174 consensus_queue.cc:1035] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:16:52.231999 26624 consensus_peers.cc:485] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): Couldn't send request to peer 89af7f423cf44717a08d6b142f0ed49d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 25b56a948668442da540ca7a198f9fbb. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:52.232224 11174 consensus_queue.cc:1035] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:52.235556 11174 raft_consensus.cc:2932] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:52.236547 29175 raft_consensus.cc:2932] T 25b56a948668442da540ca7a198f9fbb P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:52.250815 26815 raft_consensus.cc:2932] T 25b56a948668442da540ca7a198f9fbb P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:52.251709 26276 catalog_manager.cc:5484] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: config changed from index 3 to 4, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:52.313822 29175 consensus_queue.cc:237] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:16:52.316473 29306 raft_consensus.cc:1261] T 2434830a866e4f7dbe7c8fa18d48e735 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:52.316871 29574 raft_consensus.cc:1261] T 2434830a866e4f7dbe7c8fa18d48e735 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:52.317889 11267 consensus_queue.cc:1035] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:52.318284 11505 consensus_queue.cc:1035] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:16:52.320083 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): Couldn't send request to peer ba187e38a2754d60a12de9ac4856430a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2434830a866e4f7dbe7c8fa18d48e735. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:52.322510 11505 raft_consensus.cc:2932] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:16:52.326004 29306 raft_consensus.cc:2932] T 2434830a866e4f7dbe7c8fa18d48e735 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:16:52.331007 29574 raft_consensus.cc:2932] T 2434830a866e4f7dbe7c8fa18d48e735 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:16:52.336566 26280 catalog_manager.cc:5484] T 2434830a866e4f7dbe7c8fa18d48e735 P c1100e16b3b9495ca61d4f95e89b6a89 reported cstate change: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New cstate: current_term: 1 leader_uuid: "1cb67158133e4648a6eceba32d4dbba3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:16:52.444952 11799 ts_tablet_manager.cc:902] T 4bf6e7410afa4c048f5842ec2a426d77 P 1cb67158133e4648a6eceba32d4dbba3: Initiating tablet copy from peer ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:16:52.453668 26816 consensus_queue.cc:237] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:16:52.456640 26686 raft_consensus.cc:1261] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:52.456751 29438 raft_consensus.cc:1261] T ce03ee354c54457487ea939301ccd90e P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:52.457365 11504 consensus_queue.cc:1035] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:52.457665 11457 consensus_queue.cc:1035] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 14:16:52.458691 26754 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): Couldn't send request to peer 1cb67158133e4648a6eceba32d4dbba3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce03ee354c54457487ea939301ccd90e. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:52.483664 11799 tablet_copy_client.cc:308] T 4bf6e7410afa4c048f5842ec2a426d77 P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.194:36025
I20240502 14:16:52.484678 26578 tablet_copy_service.cc:140] P ba187e38a2754d60a12de9ac4856430a: Received BeginTabletCopySession request for tablet 4bf6e7410afa4c048f5842ec2a426d77 from peer 1cb67158133e4648a6eceba32d4dbba3 ({username='slave'} at 127.19.94.197:58769)
I20240502 14:16:52.484889 26578 tablet_copy_service.cc:161] P ba187e38a2754d60a12de9ac4856430a: Beginning new tablet copy session on tablet 4bf6e7410afa4c048f5842ec2a426d77 from peer 1cb67158133e4648a6eceba32d4dbba3 at {username='slave'} at 127.19.94.197:58769: session id = 1cb67158133e4648a6eceba32d4dbba3-4bf6e7410afa4c048f5842ec2a426d77
I20240502 14:16:52.488085 26578 tablet_copy_source_session.cc:205] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:16:52.491573 11799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bf6e7410afa4c048f5842ec2a426d77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:52.502334 11799 tablet_copy_client.cc:783] T 4bf6e7410afa4c048f5842ec2a426d77 P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Starting download of 5 data blocks...
I20240502 14:16:52.513307 11807 ts_tablet_manager.cc:902] T 0fc2e6c74dc34a45ae1ec46608642880 P 4092902b3e674857a0147f1b4f2cdb86: Initiating tablet copy from peer 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:16:52.521576 11799 tablet_copy_client.cc:647] T 4bf6e7410afa4c048f5842ec2a426d77 P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:52.528699 11799 tablet_copy_client.cc:523] T 4bf6e7410afa4c048f5842ec2a426d77 P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:52.534013 11457 raft_consensus.cc:2932] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:16:52.534663 29438 raft_consensus.cc:2932] T ce03ee354c54457487ea939301ccd90e P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:16:52.542032 11807 tablet_copy_client.cc:308] T 0fc2e6c74dc34a45ae1ec46608642880 P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.193:37985
I20240502 14:16:52.542979 26447 tablet_copy_service.cc:140] P 89af7f423cf44717a08d6b142f0ed49d: Received BeginTabletCopySession request for tablet 0fc2e6c74dc34a45ae1ec46608642880 from peer 4092902b3e674857a0147f1b4f2cdb86 ({username='slave'} at 127.19.94.200:60351)
I20240502 14:16:52.543146 26447 tablet_copy_service.cc:161] P 89af7f423cf44717a08d6b142f0ed49d: Beginning new tablet copy session on tablet 0fc2e6c74dc34a45ae1ec46608642880 from peer 4092902b3e674857a0147f1b4f2cdb86 at {username='slave'} at 127.19.94.200:60351: session id = 4092902b3e674857a0147f1b4f2cdb86-0fc2e6c74dc34a45ae1ec46608642880
I20240502 14:16:52.544632 11799 tablet_bootstrap.cc:492] T 4bf6e7410afa4c048f5842ec2a426d77 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:16:52.545986 26280 catalog_manager.cc:5484] T ce03ee354c54457487ea939301ccd90e P 68a3c2b17ed94e06aba66ed4178132ab reported cstate change: config changed from index 5 to 6, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:16:52.546794 26447 tablet_copy_source_session.cc:205] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:16:52.548908 11807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fc2e6c74dc34a45ae1ec46608642880. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:52.556205 11807 tablet_copy_client.cc:783] T 0fc2e6c74dc34a45ae1ec46608642880 P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Starting download of 5 data blocks...
I20240502 14:16:52.569976 26687 raft_consensus.cc:2932] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:16:52.590271 11807 tablet_copy_client.cc:647] T 0fc2e6c74dc34a45ae1ec46608642880 P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:52.594604 11807 tablet_copy_client.cc:523] T 0fc2e6c74dc34a45ae1ec46608642880 P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:52.600275 11807 tablet_bootstrap.cc:492] T 0fc2e6c74dc34a45ae1ec46608642880 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:16:52.621971 11799 tablet_bootstrap.cc:492] T 4bf6e7410afa4c048f5842ec2a426d77 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:52.622515 11799 tablet_bootstrap.cc:492] T 4bf6e7410afa4c048f5842ec2a426d77 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap complete.
I20240502 14:16:52.622855 11799 ts_tablet_manager.cc:1367] T 4bf6e7410afa4c048f5842ec2a426d77 P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.078s user 0.048s sys 0.012s
I20240502 14:16:52.623749 11799 raft_consensus.cc:348] T 4bf6e7410afa4c048f5842ec2a426d77 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:16:52.624002 11799 raft_consensus.cc:729] T 4bf6e7410afa4c048f5842ec2a426d77 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:16:52.624935 11799 consensus_queue.cc:260] T 4bf6e7410afa4c048f5842ec2a426d77 P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:16:52.625638 11799 ts_tablet_manager.cc:1398] T 4bf6e7410afa4c048f5842ec2a426d77 P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240502 14:16:52.626809 26578 tablet_copy_service.cc:342] P ba187e38a2754d60a12de9ac4856430a: Request end of tablet copy session 1cb67158133e4648a6eceba32d4dbba3-4bf6e7410afa4c048f5842ec2a426d77 received from {username='slave'} at 127.19.94.197:58769
I20240502 14:16:52.626927 26578 tablet_copy_service.cc:433] P ba187e38a2754d60a12de9ac4856430a: ending tablet copy session 1cb67158133e4648a6eceba32d4dbba3-4bf6e7410afa4c048f5842ec2a426d77 on tablet 4bf6e7410afa4c048f5842ec2a426d77 with peer 1cb67158133e4648a6eceba32d4dbba3
I20240502 14:16:52.643703 11815 ts_tablet_manager.cc:902] T 4cea6495041642f58e1eb5d26fc10df5 P c1100e16b3b9495ca61d4f95e89b6a89: Initiating tablet copy from peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:16:52.683794 11815 tablet_copy_client.cc:308] T 4cea6495041642f58e1eb5d26fc10df5 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.199:41781
I20240502 14:16:52.684592 29463 tablet_copy_service.cc:140] P 68a3c2b17ed94e06aba66ed4178132ab: Received BeginTabletCopySession request for tablet 4cea6495041642f58e1eb5d26fc10df5 from peer c1100e16b3b9495ca61d4f95e89b6a89 ({username='slave'} at 127.19.94.198:50467)
I20240502 14:16:52.684746 29463 tablet_copy_service.cc:161] P 68a3c2b17ed94e06aba66ed4178132ab: Beginning new tablet copy session on tablet 4cea6495041642f58e1eb5d26fc10df5 from peer c1100e16b3b9495ca61d4f95e89b6a89 at {username='slave'} at 127.19.94.198:50467: session id = c1100e16b3b9495ca61d4f95e89b6a89-4cea6495041642f58e1eb5d26fc10df5
I20240502 14:16:52.687376 29463 tablet_copy_source_session.cc:205] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:52.688688 11815 ts_tablet_manager.cc:1875] T 4cea6495041642f58e1eb5d26fc10df5 P c1100e16b3b9495ca61d4f95e89b6a89: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 14:16:52.691848 11807 tablet_bootstrap.cc:492] T 0fc2e6c74dc34a45ae1ec46608642880 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:52.692282 11807 tablet_bootstrap.cc:492] T 0fc2e6c74dc34a45ae1ec46608642880 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap complete.
I20240502 14:16:52.692656 11807 ts_tablet_manager.cc:1367] T 0fc2e6c74dc34a45ae1ec46608642880 P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.092s user 0.020s sys 0.014s
I20240502 14:16:52.693502 11807 raft_consensus.cc:348] T 0fc2e6c74dc34a45ae1ec46608642880 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:16:52.693785 11807 raft_consensus.cc:729] T 0fc2e6c74dc34a45ae1ec46608642880 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:16:52.694006 11807 consensus_queue.cc:260] T 0fc2e6c74dc34a45ae1ec46608642880 P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:16:52.694551 11807 ts_tablet_manager.cc:1398] T 0fc2e6c74dc34a45ae1ec46608642880 P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:16:52.696517 11815 ts_tablet_manager.cc:1888] T 4cea6495041642f58e1eb5d26fc10df5 P c1100e16b3b9495ca61d4f95e89b6a89: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.2
I20240502 14:16:52.696682 26447 tablet_copy_service.cc:342] P 89af7f423cf44717a08d6b142f0ed49d: Request end of tablet copy session 4092902b3e674857a0147f1b4f2cdb86-0fc2e6c74dc34a45ae1ec46608642880 received from {username='slave'} at 127.19.94.200:60351
I20240502 14:16:52.696800 11815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cea6495041642f58e1eb5d26fc10df5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:52.696795 26447 tablet_copy_service.cc:433] P 89af7f423cf44717a08d6b142f0ed49d: ending tablet copy session 4092902b3e674857a0147f1b4f2cdb86-0fc2e6c74dc34a45ae1ec46608642880 on tablet 0fc2e6c74dc34a45ae1ec46608642880 with peer 4092902b3e674857a0147f1b4f2cdb86
I20240502 14:16:52.700521 11815 tablet_copy_client.cc:783] T 4cea6495041642f58e1eb5d26fc10df5 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:52.700743 11815 tablet_copy_client.cc:647] T 4cea6495041642f58e1eb5d26fc10df5 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:52.705427 11815 tablet_copy_client.cc:523] T 4cea6495041642f58e1eb5d26fc10df5 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:52.712966 11815 tablet_bootstrap.cc:492] T 4cea6495041642f58e1eb5d26fc10df5 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
I20240502 14:16:52.724560 11815 tablet_bootstrap.cc:492] T 4cea6495041642f58e1eb5d26fc10df5 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:52.724968 11815 tablet_bootstrap.cc:492] T 4cea6495041642f58e1eb5d26fc10df5 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap complete.
I20240502 14:16:52.725214 11815 ts_tablet_manager.cc:1367] T 4cea6495041642f58e1eb5d26fc10df5 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20240502 14:16:52.726013 11815 raft_consensus.cc:348] T 4cea6495041642f58e1eb5d26fc10df5 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:16:52.726238 11815 raft_consensus.cc:729] T 4cea6495041642f58e1eb5d26fc10df5 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:16:52.726475 11815 consensus_queue.cc:260] T 4cea6495041642f58e1eb5d26fc10df5 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:16:52.730720 11815 ts_tablet_manager.cc:1398] T 4cea6495041642f58e1eb5d26fc10df5 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:16:52.732163 29463 tablet_copy_service.cc:342] P 68a3c2b17ed94e06aba66ed4178132ab: Request end of tablet copy session c1100e16b3b9495ca61d4f95e89b6a89-4cea6495041642f58e1eb5d26fc10df5 received from {username='slave'} at 127.19.94.198:50467
I20240502 14:16:52.732295 29463 tablet_copy_service.cc:433] P 68a3c2b17ed94e06aba66ed4178132ab: ending tablet copy session c1100e16b3b9495ca61d4f95e89b6a89-4cea6495041642f58e1eb5d26fc10df5 on tablet 4cea6495041642f58e1eb5d26fc10df5 with peer c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:16:52.739552 26425 consensus_queue.cc:237] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:16:52.740888 11837 ts_tablet_manager.cc:902] T 25b56a948668442da540ca7a198f9fbb P 89af7f423cf44717a08d6b142f0ed49d: Initiating tablet copy from peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:16:52.746435 29222 maintenance_manager.cc:419] P 1cb67158133e4648a6eceba32d4dbba3: Scheduling UndoDeltaBlockGCOp(4bf6e7410afa4c048f5842ec2a426d77): 4586 bytes on disk
I20240502 14:16:52.748669 29113 maintenance_manager.cc:643] P 1cb67158133e4648a6eceba32d4dbba3: UndoDeltaBlockGCOp(4bf6e7410afa4c048f5842ec2a426d77) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":140,"lbm_reads_lt_1ms":4}
I20240502 14:16:52.750447 26557 raft_consensus.cc:1261] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:52.751458 11490 consensus_queue.cc:1035] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:52.754931 29175 raft_consensus.cc:1261] T 9f3cbedfa1da42698615a544d9dc3276 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:52.755885 11696 consensus_queue.cc:1035] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 14:16:52.758903 26363 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9f3cbedfa1da42698615a544d9dc3276. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:52.769032 11837 tablet_copy_client.cc:308] T 25b56a948668442da540ca7a198f9fbb P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.195:44325
I20240502 14:16:52.769788 26709 tablet_copy_service.cc:140] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Received BeginTabletCopySession request for tablet 25b56a948668442da540ca7a198f9fbb from peer 89af7f423cf44717a08d6b142f0ed49d ({username='slave'} at 127.19.94.193:35337)
I20240502 14:16:52.769980 26709 tablet_copy_service.cc:161] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Beginning new tablet copy session on tablet 25b56a948668442da540ca7a198f9fbb from peer 89af7f423cf44717a08d6b142f0ed49d at {username='slave'} at 127.19.94.193:35337: session id = 89af7f423cf44717a08d6b142f0ed49d-25b56a948668442da540ca7a198f9fbb
I20240502 14:16:52.770993 11696 raft_consensus.cc:2932] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:52.773200 26709 tablet_copy_source_session.cc:205] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:52.775009 29175 raft_consensus.cc:2932] T 9f3cbedfa1da42698615a544d9dc3276 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:52.776233 11837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25b56a948668442da540ca7a198f9fbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:52.785099 11837 tablet_copy_client.cc:783] T 25b56a948668442da540ca7a198f9fbb P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:52.785317 11837 tablet_copy_client.cc:647] T 25b56a948668442da540ca7a198f9fbb P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:52.786816 11837 tablet_copy_client.cc:523] T 25b56a948668442da540ca7a198f9fbb P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:52.789168 26556 raft_consensus.cc:2932] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:52.790352 11837 tablet_bootstrap.cc:492] T 25b56a948668442da540ca7a198f9fbb P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:16:52.811722 11837 tablet_bootstrap.cc:492] T 25b56a948668442da540ca7a198f9fbb P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:52.812249 11837 tablet_bootstrap.cc:492] T 25b56a948668442da540ca7a198f9fbb P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap complete.
I20240502 14:16:52.812592 11837 ts_tablet_manager.cc:1367] T 25b56a948668442da540ca7a198f9fbb P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.022s user 0.017s sys 0.000s
I20240502 14:16:52.813692 11837 raft_consensus.cc:348] T 25b56a948668442da540ca7a198f9fbb P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:52.813870 11837 raft_consensus.cc:729] T 25b56a948668442da540ca7a198f9fbb P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:16:52.814066 11837 consensus_queue.cc:260] T 25b56a948668442da540ca7a198f9fbb P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:52.814594 11837 ts_tablet_manager.cc:1398] T 25b56a948668442da540ca7a198f9fbb P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:16:52.820786 26280 catalog_manager.cc:5484] T 9f3cbedfa1da42698615a544d9dc3276 P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: config changed from index 5 to 6, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:52.821825 26709 tablet_copy_service.cc:342] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Request end of tablet copy session 89af7f423cf44717a08d6b142f0ed49d-25b56a948668442da540ca7a198f9fbb received from {username='slave'} at 127.19.94.193:35337
I20240502 14:16:52.821980 26709 tablet_copy_service.cc:433] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: ending tablet copy session 89af7f423cf44717a08d6b142f0ed49d-25b56a948668442da540ca7a198f9fbb on tablet 25b56a948668442da540ca7a198f9fbb with peer 89af7f423cf44717a08d6b142f0ed49d
I20240502 14:16:52.871367 26556 consensus_queue.cc:237] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:16:52.874719 29175 raft_consensus.cc:1261] T 9d28bfbd29c94d2ebbedbae53231bf1e P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:52.875017 29306 raft_consensus.cc:1261] T 9d28bfbd29c94d2ebbedbae53231bf1e P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:16:52.875391 26494 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): Couldn't send request to peer 69ab78b2a1b74d2bbf2cdeafa0a4af23. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9d28bfbd29c94d2ebbedbae53231bf1e. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:52.875397 11720 consensus_queue.cc:1035] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:52.875774 11516 consensus_queue.cc:1035] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:52.881731 11720 raft_consensus.cc:2932] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:52.882371 29175 raft_consensus.cc:2932] T 9d28bfbd29c94d2ebbedbae53231bf1e P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:52.889418 26276 catalog_manager.cc:5484] T 9d28bfbd29c94d2ebbedbae53231bf1e P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:52.896765 29306 raft_consensus.cc:2932] T 9d28bfbd29c94d2ebbedbae53231bf1e P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } }
I20240502 14:16:52.902235 29621 maintenance_manager.cc:419] P 4092902b3e674857a0147f1b4f2cdb86: Scheduling UndoDeltaBlockGCOp(0fc2e6c74dc34a45ae1ec46608642880): 4840 bytes on disk
I20240502 14:16:52.903879 29512 maintenance_manager.cc:643] P 4092902b3e674857a0147f1b4f2cdb86: UndoDeltaBlockGCOp(0fc2e6c74dc34a45ae1ec46608642880) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":155,"lbm_reads_lt_1ms":4}
I20240502 14:16:52.912978 29575 raft_consensus.cc:1203] T 0fc2e6c74dc34a45ae1ec46608642880 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 14:16:52.929363 11874 ts_tablet_manager.cc:902] T 2434830a866e4f7dbe7c8fa18d48e735 P ba187e38a2754d60a12de9ac4856430a: Initiating tablet copy from peer 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:16:52.937218 11874 tablet_copy_client.cc:308] T 2434830a866e4f7dbe7c8fa18d48e735 P ba187e38a2754d60a12de9ac4856430a: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.197:38715
I20240502 14:16:52.938005 29196 tablet_copy_service.cc:140] P 1cb67158133e4648a6eceba32d4dbba3: Received BeginTabletCopySession request for tablet 2434830a866e4f7dbe7c8fa18d48e735 from peer ba187e38a2754d60a12de9ac4856430a ({username='slave'} at 127.19.94.194:50529)
I20240502 14:16:52.938215 29196 tablet_copy_service.cc:161] P 1cb67158133e4648a6eceba32d4dbba3: Beginning new tablet copy session on tablet 2434830a866e4f7dbe7c8fa18d48e735 from peer ba187e38a2754d60a12de9ac4856430a at {username='slave'} at 127.19.94.194:50529: session id = ba187e38a2754d60a12de9ac4856430a-2434830a866e4f7dbe7c8fa18d48e735
I20240502 14:16:52.940500 29196 tablet_copy_source_session.cc:205] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:52.941857 11874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2434830a866e4f7dbe7c8fa18d48e735. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:52.952096 11874 tablet_copy_client.cc:783] T 2434830a866e4f7dbe7c8fa18d48e735 P ba187e38a2754d60a12de9ac4856430a: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:52.952757 11874 tablet_copy_client.cc:647] T 2434830a866e4f7dbe7c8fa18d48e735 P ba187e38a2754d60a12de9ac4856430a: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:52.955073 11874 tablet_copy_client.cc:523] T 2434830a866e4f7dbe7c8fa18d48e735 P ba187e38a2754d60a12de9ac4856430a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:52.959331 11874 tablet_bootstrap.cc:492] T 2434830a866e4f7dbe7c8fa18d48e735 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:16:52.964514 29175 raft_consensus.cc:1203] T 4bf6e7410afa4c048f5842ec2a426d77 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 14:16:52.966809 11874 tablet_bootstrap.cc:492] T 2434830a866e4f7dbe7c8fa18d48e735 P ba187e38a2754d60a12de9ac4856430a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:52.967221 11874 tablet_bootstrap.cc:492] T 2434830a866e4f7dbe7c8fa18d48e735 P ba187e38a2754d60a12de9ac4856430a: Bootstrap complete.
I20240502 14:16:52.967880 11874 ts_tablet_manager.cc:1367] T 2434830a866e4f7dbe7c8fa18d48e735 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.002s
I20240502 14:16:52.968737 11874 raft_consensus.cc:348] T 2434830a866e4f7dbe7c8fa18d48e735 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:16:52.969321 11874 raft_consensus.cc:729] T 2434830a866e4f7dbe7c8fa18d48e735 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:16:52.969626 11874 consensus_queue.cc:260] T 2434830a866e4f7dbe7c8fa18d48e735 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:16:52.970196 11874 ts_tablet_manager.cc:1398] T 2434830a866e4f7dbe7c8fa18d48e735 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:16:52.971140 29196 tablet_copy_service.cc:342] P 1cb67158133e4648a6eceba32d4dbba3: Request end of tablet copy session ba187e38a2754d60a12de9ac4856430a-2434830a866e4f7dbe7c8fa18d48e735 received from {username='slave'} at 127.19.94.194:50529
I20240502 14:16:52.971283 29196 tablet_copy_service.cc:433] P 1cb67158133e4648a6eceba32d4dbba3: ending tablet copy session ba187e38a2754d60a12de9ac4856430a-2434830a866e4f7dbe7c8fa18d48e735 on tablet 2434830a866e4f7dbe7c8fa18d48e735 with peer ba187e38a2754d60a12de9ac4856430a
I20240502 14:16:52.972153 26425 consensus_queue.cc:237] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
W20240502 14:16:52.974963 26363 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097): Couldn't send request to peer 4092902b3e674857a0147f1b4f2cdb86. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bb3e4ec75888419aa4e065cf5513ae5b. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:52.975032 29306 raft_consensus.cc:1261] T bb3e4ec75888419aa4e065cf5513ae5b P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:52.976327 11696 consensus_queue.cc:1035] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:52.976569 26816 raft_consensus.cc:1261] T bb3e4ec75888419aa4e065cf5513ae5b P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:52.977247 11696 consensus_queue.cc:1035] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:52.980237 11696 raft_consensus.cc:2932] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:16:52.980928 26816 raft_consensus.cc:2932] T bb3e4ec75888419aa4e065cf5513ae5b P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:16:52.985459 29309 raft_consensus.cc:2932] T bb3e4ec75888419aa4e065cf5513ae5b P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:16:52.989342 26280 catalog_manager.cc:5484] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: config changed from index 5 to 6, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:16:53.043887 29575 consensus_queue.cc:237] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:16:53.046748 29438 raft_consensus.cc:1261] T 82e51aa5371e4731bc00daafc65c7a88 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:53.047495 11888 consensus_queue.cc:1035] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:53.049751 29175 raft_consensus.cc:1261] T 82e51aa5371e4731bc00daafc65c7a88 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:16:53.050601 11888 consensus_queue.cc:1035] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:16:53.053344 29510 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): Couldn't send request to peer ba187e38a2754d60a12de9ac4856430a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 82e51aa5371e4731bc00daafc65c7a88. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:53.057317 11158 raft_consensus.cc:2932] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:16:53.057920 29175 raft_consensus.cc:2932] T 82e51aa5371e4731bc00daafc65c7a88 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:16:53.063995 29438 raft_consensus.cc:2932] T 82e51aa5371e4731bc00daafc65c7a88 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:16:53.067245 26280 catalog_manager.cc:5484] T 82e51aa5371e4731bc00daafc65c7a88 P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: config changed from index -1 to 2, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) added. New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:16:53.079162 11799 ts_tablet_manager.cc:902] T ce03ee354c54457487ea939301ccd90e P 1cb67158133e4648a6eceba32d4dbba3: Initiating tablet copy from peer e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:16:53.101229 11799 tablet_copy_client.cc:308] T ce03ee354c54457487ea939301ccd90e P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.196:32843
I20240502 14:16:53.102196 26839 tablet_copy_service.cc:140] P e4cfb8fc5a474177af6848a6efa13a27: Received BeginTabletCopySession request for tablet ce03ee354c54457487ea939301ccd90e from peer 1cb67158133e4648a6eceba32d4dbba3 ({username='slave'} at 127.19.94.197:46803)
I20240502 14:16:53.102416 26839 tablet_copy_service.cc:161] P e4cfb8fc5a474177af6848a6efa13a27: Beginning new tablet copy session on tablet ce03ee354c54457487ea939301ccd90e from peer 1cb67158133e4648a6eceba32d4dbba3 at {username='slave'} at 127.19.94.197:46803: session id = 1cb67158133e4648a6eceba32d4dbba3-ce03ee354c54457487ea939301ccd90e
I20240502 14:16:53.106619 26839 tablet_copy_source_session.cc:205] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:16:53.108156 11799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce03ee354c54457487ea939301ccd90e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:53.115921 11799 tablet_copy_client.cc:783] T ce03ee354c54457487ea939301ccd90e P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Starting download of 5 data blocks...
I20240502 14:16:53.136354 11799 tablet_copy_client.cc:647] T ce03ee354c54457487ea939301ccd90e P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:53.140697 11799 tablet_copy_client.cc:523] T ce03ee354c54457487ea939301ccd90e P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:53.146010 11799 tablet_bootstrap.cc:492] T ce03ee354c54457487ea939301ccd90e P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
I20240502 14:16:53.180307 26425 raft_consensus.cc:1203] T 25b56a948668442da540ca7a198f9fbb P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:16:53.185557 11799 tablet_bootstrap.cc:492] T ce03ee354c54457487ea939301ccd90e P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:53.186004 11799 tablet_bootstrap.cc:492] T ce03ee354c54457487ea939301ccd90e P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap complete.
I20240502 14:16:53.186312 11799 ts_tablet_manager.cc:1367] T ce03ee354c54457487ea939301ccd90e P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.040s user 0.020s sys 0.000s
I20240502 14:16:53.187047 11799 raft_consensus.cc:348] T ce03ee354c54457487ea939301ccd90e P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:16:53.187315 11799 raft_consensus.cc:729] T ce03ee354c54457487ea939301ccd90e P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:16:53.187572 11799 consensus_queue.cc:260] T ce03ee354c54457487ea939301ccd90e P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:16:53.188125 11799 ts_tablet_manager.cc:1398] T ce03ee354c54457487ea939301ccd90e P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:16:53.189054 26839 tablet_copy_service.cc:342] P e4cfb8fc5a474177af6848a6efa13a27: Request end of tablet copy session 1cb67158133e4648a6eceba32d4dbba3-ce03ee354c54457487ea939301ccd90e received from {username='slave'} at 127.19.94.197:46803
I20240502 14:16:53.189199 26839 tablet_copy_service.cc:433] P e4cfb8fc5a474177af6848a6efa13a27: ending tablet copy session 1cb67158133e4648a6eceba32d4dbba3-ce03ee354c54457487ea939301ccd90e on tablet ce03ee354c54457487ea939301ccd90e with peer 1cb67158133e4648a6eceba32d4dbba3
I20240502 14:16:53.204823 11936 ts_tablet_manager.cc:902] T 9f3cbedfa1da42698615a544d9dc3276 P 68a3c2b17ed94e06aba66ed4178132ab: Initiating tablet copy from peer 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:16:53.215620 29309 raft_consensus.cc:1203] T 4cea6495041642f58e1eb5d26fc10df5 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:16:53.224004 26816 consensus_queue.cc:237] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:16:53.226784 26557 raft_consensus.cc:1261] T a42e68ab03f84d4b92893006fcf6f229 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:16:53.227365 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a42e68ab03f84d4b92893006fcf6f229. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:53.228816 26425 raft_consensus.cc:1261] T a42e68ab03f84d4b92893006fcf6f229 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:53.229192 11457 consensus_queue.cc:1035] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:53.230330 11457 consensus_queue.cc:1035] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:53.235913 11457 raft_consensus.cc:2932] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:53.239289 26425 raft_consensus.cc:2932] T a42e68ab03f84d4b92893006fcf6f229 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:53.254050 26280 catalog_manager.cc:5484] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: config changed from index 3 to 4, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:53.254925 26557 raft_consensus.cc:2932] T a42e68ab03f84d4b92893006fcf6f229 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:16:53.275027 29222 maintenance_manager.cc:419] P 1cb67158133e4648a6eceba32d4dbba3: Scheduling FlushMRSOp(0fc2e6c74dc34a45ae1ec46608642880): perf score=0.033388
I20240502 14:16:53.283217 11936 tablet_copy_client.cc:308] T 9f3cbedfa1da42698615a544d9dc3276 P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.193:37985
I20240502 14:16:53.284543 26447 tablet_copy_service.cc:140] P 89af7f423cf44717a08d6b142f0ed49d: Received BeginTabletCopySession request for tablet 9f3cbedfa1da42698615a544d9dc3276 from peer 68a3c2b17ed94e06aba66ed4178132ab ({username='slave'} at 127.19.94.199:39405)
I20240502 14:16:53.284802 26447 tablet_copy_service.cc:161] P 89af7f423cf44717a08d6b142f0ed49d: Beginning new tablet copy session on tablet 9f3cbedfa1da42698615a544d9dc3276 from peer 68a3c2b17ed94e06aba66ed4178132ab at {username='slave'} at 127.19.94.199:39405: session id = 68a3c2b17ed94e06aba66ed4178132ab-9f3cbedfa1da42698615a544d9dc3276
I20240502 14:16:53.288833 26447 tablet_copy_source_session.cc:205] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:16:53.292569 11936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f3cbedfa1da42698615a544d9dc3276. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:53.308182 11936 tablet_copy_client.cc:783] T 9f3cbedfa1da42698615a544d9dc3276 P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Starting download of 5 data blocks...
I20240502 14:16:53.346119 11950 ts_tablet_manager.cc:902] T 9d28bfbd29c94d2ebbedbae53231bf1e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Initiating tablet copy from peer ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:16:53.348628 11936 tablet_copy_client.cc:647] T 9f3cbedfa1da42698615a544d9dc3276 P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:53.353812 11936 tablet_copy_client.cc:523] T 9f3cbedfa1da42698615a544d9dc3276 P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:53.360221 11936 tablet_bootstrap.cc:492] T 9f3cbedfa1da42698615a544d9dc3276 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:16:53.408519 11950 tablet_copy_client.cc:308] T 9d28bfbd29c94d2ebbedbae53231bf1e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.194:36025
I20240502 14:16:53.409245 26578 tablet_copy_service.cc:140] P ba187e38a2754d60a12de9ac4856430a: Received BeginTabletCopySession request for tablet 9d28bfbd29c94d2ebbedbae53231bf1e from peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 ({username='slave'} at 127.19.94.195:34463)
I20240502 14:16:53.409428 26578 tablet_copy_service.cc:161] P ba187e38a2754d60a12de9ac4856430a: Beginning new tablet copy session on tablet 9d28bfbd29c94d2ebbedbae53231bf1e from peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 at {username='slave'} at 127.19.94.195:34463: session id = 69ab78b2a1b74d2bbf2cdeafa0a4af23-9d28bfbd29c94d2ebbedbae53231bf1e
I20240502 14:16:53.411900 26578 tablet_copy_source_session.cc:205] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:53.413709 11950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d28bfbd29c94d2ebbedbae53231bf1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:53.428541 11874 ts_tablet_manager.cc:902] T 82e51aa5371e4731bc00daafc65c7a88 P ba187e38a2754d60a12de9ac4856430a: Initiating tablet copy from peer 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:16:53.431250 11950 tablet_copy_client.cc:783] T 9d28bfbd29c94d2ebbedbae53231bf1e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:53.431712 11950 tablet_copy_client.cc:647] T 9d28bfbd29c94d2ebbedbae53231bf1e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:53.433832 11950 tablet_copy_client.cc:523] T 9d28bfbd29c94d2ebbedbae53231bf1e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:53.439016 11936 tablet_bootstrap.cc:492] T 9f3cbedfa1da42698615a544d9dc3276 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:53.439482 11936 tablet_bootstrap.cc:492] T 9f3cbedfa1da42698615a544d9dc3276 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap complete.
I20240502 14:16:53.439803 11936 ts_tablet_manager.cc:1367] T 9f3cbedfa1da42698615a544d9dc3276 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.080s user 0.036s sys 0.002s
I20240502 14:16:53.440567 11936 raft_consensus.cc:348] T 9f3cbedfa1da42698615a544d9dc3276 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:16:53.440797 11936 raft_consensus.cc:729] T 9f3cbedfa1da42698615a544d9dc3276 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:16:53.444056 11950 tablet_bootstrap.cc:492] T 9d28bfbd29c94d2ebbedbae53231bf1e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap starting.
I20240502 14:16:53.447320 11936 consensus_queue.cc:260] T 9f3cbedfa1da42698615a544d9dc3276 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:16:53.462313 11874 tablet_copy_client.cc:308] T 82e51aa5371e4731bc00daafc65c7a88 P ba187e38a2754d60a12de9ac4856430a: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.200:46097
I20240502 14:16:53.463209 29594 tablet_copy_service.cc:140] P 4092902b3e674857a0147f1b4f2cdb86: Received BeginTabletCopySession request for tablet 82e51aa5371e4731bc00daafc65c7a88 from peer ba187e38a2754d60a12de9ac4856430a ({username='slave'} at 127.19.94.194:37897)
I20240502 14:16:53.463418 29594 tablet_copy_service.cc:161] P 4092902b3e674857a0147f1b4f2cdb86: Beginning new tablet copy session on tablet 82e51aa5371e4731bc00daafc65c7a88 from peer ba187e38a2754d60a12de9ac4856430a at {username='slave'} at 127.19.94.194:37897: session id = ba187e38a2754d60a12de9ac4856430a-82e51aa5371e4731bc00daafc65c7a88
I20240502 14:16:53.467618 29594 tablet_copy_source_session.cc:205] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:53.463268 11964 ts_tablet_manager.cc:902] T bb3e4ec75888419aa4e065cf5513ae5b P 4092902b3e674857a0147f1b4f2cdb86: Initiating tablet copy from peer 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:16:53.469265 11874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82e51aa5371e4731bc00daafc65c7a88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:53.474313 11936 ts_tablet_manager.cc:1398] T 9f3cbedfa1da42698615a544d9dc3276 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.034s user 0.003s sys 0.000s
I20240502 14:16:53.477480 29175 raft_consensus.cc:1203] T ce03ee354c54457487ea939301ccd90e P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 14:16:53.487108 29489 maintenance_manager.cc:419] P 68a3c2b17ed94e06aba66ed4178132ab: Scheduling UndoDeltaBlockGCOp(9f3cbedfa1da42698615a544d9dc3276): 4925 bytes on disk
I20240502 14:16:53.487511 11874 tablet_copy_client.cc:783] T 82e51aa5371e4731bc00daafc65c7a88 P ba187e38a2754d60a12de9ac4856430a: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:53.487797 11874 tablet_copy_client.cc:647] T 82e51aa5371e4731bc00daafc65c7a88 P ba187e38a2754d60a12de9ac4856430a: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:53.489149 29380 maintenance_manager.cc:643] P 68a3c2b17ed94e06aba66ed4178132ab: UndoDeltaBlockGCOp(9f3cbedfa1da42698615a544d9dc3276) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":136,"lbm_reads_lt_1ms":4}
I20240502 14:16:53.492477 26447 tablet_copy_service.cc:342] P 89af7f423cf44717a08d6b142f0ed49d: Request end of tablet copy session 68a3c2b17ed94e06aba66ed4178132ab-9f3cbedfa1da42698615a544d9dc3276 received from {username='slave'} at 127.19.94.199:39405
I20240502 14:16:53.492609 26447 tablet_copy_service.cc:433] P 89af7f423cf44717a08d6b142f0ed49d: ending tablet copy session 68a3c2b17ed94e06aba66ed4178132ab-9f3cbedfa1da42698615a544d9dc3276 on tablet 9f3cbedfa1da42698615a544d9dc3276 with peer 68a3c2b17ed94e06aba66ed4178132ab
I20240502 14:16:53.500553 11964 tablet_copy_client.cc:308] T bb3e4ec75888419aa4e065cf5513ae5b P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.193:37985
I20240502 14:16:53.501257 26447 tablet_copy_service.cc:140] P 89af7f423cf44717a08d6b142f0ed49d: Received BeginTabletCopySession request for tablet bb3e4ec75888419aa4e065cf5513ae5b from peer 4092902b3e674857a0147f1b4f2cdb86 ({username='slave'} at 127.19.94.200:60351)
I20240502 14:16:53.501421 26447 tablet_copy_service.cc:161] P 89af7f423cf44717a08d6b142f0ed49d: Beginning new tablet copy session on tablet bb3e4ec75888419aa4e065cf5513ae5b from peer 4092902b3e674857a0147f1b4f2cdb86 at {username='slave'} at 127.19.94.200:60351: session id = 4092902b3e674857a0147f1b4f2cdb86-bb3e4ec75888419aa4e065cf5513ae5b
I20240502 14:16:53.502753 11874 tablet_copy_client.cc:523] T 82e51aa5371e4731bc00daafc65c7a88 P ba187e38a2754d60a12de9ac4856430a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:53.505007 26447 tablet_copy_source_session.cc:205] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:16:53.506620 11964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb3e4ec75888419aa4e065cf5513ae5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:53.514118 11874 tablet_bootstrap.cc:492] T 82e51aa5371e4731bc00daafc65c7a88 P ba187e38a2754d60a12de9ac4856430a: Bootstrap starting.
I20240502 14:16:53.520648 11950 tablet_bootstrap.cc:492] T 9d28bfbd29c94d2ebbedbae53231bf1e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:53.521142 11950 tablet_bootstrap.cc:492] T 9d28bfbd29c94d2ebbedbae53231bf1e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Bootstrap complete.
I20240502 14:16:53.521494 11950 ts_tablet_manager.cc:1367] T 9d28bfbd29c94d2ebbedbae53231bf1e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent bootstrapping tablet: real 0.078s user 0.009s sys 0.012s
I20240502 14:16:53.522555 11950 raft_consensus.cc:348] T 9d28bfbd29c94d2ebbedbae53231bf1e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:16:53.523023 11950 raft_consensus.cc:729] T 9d28bfbd29c94d2ebbedbae53231bf1e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Initialized, Role: FOLLOWER
I20240502 14:16:53.523345 11950 consensus_queue.cc:260] T 9d28bfbd29c94d2ebbedbae53231bf1e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:16:53.529978 11874 tablet_bootstrap.cc:492] T 82e51aa5371e4731bc00daafc65c7a88 P ba187e38a2754d60a12de9ac4856430a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:53.530309 11950 ts_tablet_manager.cc:1398] T 9d28bfbd29c94d2ebbedbae53231bf1e P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 14:16:53.530443 11874 tablet_bootstrap.cc:492] T 82e51aa5371e4731bc00daafc65c7a88 P ba187e38a2754d60a12de9ac4856430a: Bootstrap complete.
I20240502 14:16:53.530745 11874 ts_tablet_manager.cc:1367] T 82e51aa5371e4731bc00daafc65c7a88 P ba187e38a2754d60a12de9ac4856430a: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.001s
I20240502 14:16:53.532073 26578 tablet_copy_service.cc:342] P ba187e38a2754d60a12de9ac4856430a: Request end of tablet copy session 69ab78b2a1b74d2bbf2cdeafa0a4af23-9d28bfbd29c94d2ebbedbae53231bf1e received from {username='slave'} at 127.19.94.195:34463
I20240502 14:16:53.532869 26578 tablet_copy_service.cc:433] P ba187e38a2754d60a12de9ac4856430a: ending tablet copy session 69ab78b2a1b74d2bbf2cdeafa0a4af23-9d28bfbd29c94d2ebbedbae53231bf1e on tablet 9d28bfbd29c94d2ebbedbae53231bf1e with peer 69ab78b2a1b74d2bbf2cdeafa0a4af23
I20240502 14:16:53.531518 11874 raft_consensus.cc:348] T 82e51aa5371e4731bc00daafc65c7a88 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:16:53.535151 11874 raft_consensus.cc:729] T 82e51aa5371e4731bc00daafc65c7a88 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Initialized, Role: FOLLOWER
I20240502 14:16:53.535493 11874 consensus_queue.cc:260] T 82e51aa5371e4731bc00daafc65c7a88 P ba187e38a2754d60a12de9ac4856430a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:16:53.536944 11874 ts_tablet_manager.cc:1398] T 82e51aa5371e4731bc00daafc65c7a88 P ba187e38a2754d60a12de9ac4856430a: Time spent starting tablet: real 0.006s user 0.003s sys 0.001s
I20240502 14:16:53.537364 26556 raft_consensus.cc:1203] T 2434830a866e4f7dbe7c8fa18d48e735 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:16:53.538350 29594 tablet_copy_service.cc:342] P 4092902b3e674857a0147f1b4f2cdb86: Request end of tablet copy session ba187e38a2754d60a12de9ac4856430a-82e51aa5371e4731bc00daafc65c7a88 received from {username='slave'} at 127.19.94.194:37897
I20240502 14:16:53.538483 29594 tablet_copy_service.cc:433] P 4092902b3e674857a0147f1b4f2cdb86: ending tablet copy session ba187e38a2754d60a12de9ac4856430a-82e51aa5371e4731bc00daafc65c7a88 on tablet 82e51aa5371e4731bc00daafc65c7a88 with peer ba187e38a2754d60a12de9ac4856430a
I20240502 14:16:53.545497 11964 tablet_copy_client.cc:783] T bb3e4ec75888419aa4e065cf5513ae5b P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Starting download of 5 data blocks...
I20240502 14:16:53.553740 29355 maintenance_manager.cc:419] P c1100e16b3b9495ca61d4f95e89b6a89: Scheduling FlushMRSOp(bb3e4ec75888419aa4e065cf5513ae5b): perf score=0.033336
I20240502 14:16:53.555572 11964 tablet_copy_client.cc:647] T bb3e4ec75888419aa4e065cf5513ae5b P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:53.560199 11964 tablet_copy_client.cc:523] T bb3e4ec75888419aa4e065cf5513ae5b P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:53.567569 11964 tablet_bootstrap.cc:492] T bb3e4ec75888419aa4e065cf5513ae5b P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
I20240502 14:16:53.573163 26556 consensus_queue.cc:237] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
W20240502 14:16:53.599136 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 310aa6ad3604487589dbe96afa25bb05. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:53.599390 26686 raft_consensus.cc:1261] T 310aa6ad3604487589dbe96afa25bb05 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:53.599733 29175 raft_consensus.cc:1261] T 310aa6ad3604487589dbe96afa25bb05 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:53.600296 11720 consensus_queue.cc:1035] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:53.601717 11720 consensus_queue.cc:1035] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:53.627414 29113 maintenance_manager.cc:643] P 1cb67158133e4648a6eceba32d4dbba3: FlushMRSOp(0fc2e6c74dc34a45ae1ec46608642880) complete. Timing: real 0.352s user 0.064s sys 0.017s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":11437,"dirs.queue_time_us":7937,"dirs.run_cpu_time_us":322,"dirs.run_wall_time_us":3418,"drs_written":1,"lbm_read_time_us":18080,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":3,"lbm_write_time_us":7419,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":21,"peak_mem_usage":0,"rows_written":676,"thread_start_us":37466,"threads_started":3,"wal-append.queue_time_us":7016}
I20240502 14:16:53.630867 29222 maintenance_manager.cc:419] P 1cb67158133e4648a6eceba32d4dbba3: Scheduling UndoDeltaBlockGCOp(ce03ee354c54457487ea939301ccd90e): 4925 bytes on disk
I20240502 14:16:53.642189 29438 raft_consensus.cc:1203] T 9f3cbedfa1da42698615a544d9dc3276 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 14:16:53.645978 29113 maintenance_manager.cc:643] P 1cb67158133e4648a6eceba32d4dbba3: UndoDeltaBlockGCOp(ce03ee354c54457487ea939301ccd90e) complete. Timing: real 0.004s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":130,"lbm_reads_lt_1ms":4}
I20240502 14:16:53.647367 11516 raft_consensus.cc:2932] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:53.648700 29175 raft_consensus.cc:2932] T 310aa6ad3604487589dbe96afa25bb05 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:53.650362 29222 maintenance_manager.cc:419] P 1cb67158133e4648a6eceba32d4dbba3: Scheduling UndoDeltaBlockGCOp(0fc2e6c74dc34a45ae1ec46608642880): 4840 bytes on disk
I20240502 14:16:53.657598 26280 catalog_manager.cc:5484] T 310aa6ad3604487589dbe96afa25bb05 P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: config changed from index 5 to 6, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:53.664873 29113 maintenance_manager.cc:643] P 1cb67158133e4648a6eceba32d4dbba3: UndoDeltaBlockGCOp(0fc2e6c74dc34a45ae1ec46608642880) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":163,"lbm_reads_lt_1ms":4}
I20240502 14:16:53.663914 26687 raft_consensus.cc:2932] T 310aa6ad3604487589dbe96afa25bb05 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:53.709573 11964 tablet_bootstrap.cc:492] T bb3e4ec75888419aa4e065cf5513ae5b P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:53.712960 11964 tablet_bootstrap.cc:492] T bb3e4ec75888419aa4e065cf5513ae5b P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap complete.
I20240502 14:16:53.714027 11964 ts_tablet_manager.cc:1367] T bb3e4ec75888419aa4e065cf5513ae5b P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.147s user 0.024s sys 0.011s
I20240502 14:16:53.715869 11964 raft_consensus.cc:348] T bb3e4ec75888419aa4e065cf5513ae5b P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:16:53.719152 11964 raft_consensus.cc:729] T bb3e4ec75888419aa4e065cf5513ae5b P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:16:53.719724 11964 consensus_queue.cc:260] T bb3e4ec75888419aa4e065cf5513ae5b P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:16:53.749316 11964 ts_tablet_manager.cc:1398] T bb3e4ec75888419aa4e065cf5513ae5b P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.034s user 0.002s sys 0.000s
I20240502 14:16:53.750597 26447 tablet_copy_service.cc:342] P 89af7f423cf44717a08d6b142f0ed49d: Request end of tablet copy session 4092902b3e674857a0147f1b4f2cdb86-bb3e4ec75888419aa4e065cf5513ae5b received from {username='slave'} at 127.19.94.200:60351
I20240502 14:16:53.750747 26447 tablet_copy_service.cc:433] P 89af7f423cf44717a08d6b142f0ed49d: ending tablet copy session 4092902b3e674857a0147f1b4f2cdb86-bb3e4ec75888419aa4e065cf5513ae5b on tablet bb3e4ec75888419aa4e065cf5513ae5b with peer 4092902b3e674857a0147f1b4f2cdb86
I20240502 14:16:53.824492 29246 maintenance_manager.cc:643] P c1100e16b3b9495ca61d4f95e89b6a89: FlushMRSOp(bb3e4ec75888419aa4e065cf5513ae5b) complete. Timing: real 0.269s user 0.075s sys 0.028s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":34509,"dirs.queue_time_us":12039,"dirs.run_cpu_time_us":250,"dirs.run_wall_time_us":7409,"drs_written":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4872,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":21,"peak_mem_usage":0,"rows_written":689,"thread_start_us":46708,"threads_started":2}
I20240502 14:16:53.828281 29355 maintenance_manager.cc:419] P c1100e16b3b9495ca61d4f95e89b6a89: Scheduling UndoDeltaBlockGCOp(bb3e4ec75888419aa4e065cf5513ae5b): 4925 bytes on disk
I20240502 14:16:53.829640 29246 maintenance_manager.cc:643] P c1100e16b3b9495ca61d4f95e89b6a89: UndoDeltaBlockGCOp(bb3e4ec75888419aa4e065cf5513ae5b) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":118,"lbm_reads_lt_1ms":4}
I20240502 14:16:53.837882 26687 raft_consensus.cc:1203] T 9d28bfbd29c94d2ebbedbae53231bf1e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:16:53.848832 11936 ts_tablet_manager.cc:902] T a42e68ab03f84d4b92893006fcf6f229 P 68a3c2b17ed94e06aba66ed4178132ab: Initiating tablet copy from peer e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:16:53.875409 11936 tablet_copy_client.cc:308] T a42e68ab03f84d4b92893006fcf6f229 P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.196:32843
I20240502 14:16:53.876525 26839 tablet_copy_service.cc:140] P e4cfb8fc5a474177af6848a6efa13a27: Received BeginTabletCopySession request for tablet a42e68ab03f84d4b92893006fcf6f229 from peer 68a3c2b17ed94e06aba66ed4178132ab ({username='slave'} at 127.19.94.199:38699)
I20240502 14:16:53.876736 26839 tablet_copy_service.cc:161] P e4cfb8fc5a474177af6848a6efa13a27: Beginning new tablet copy session on tablet a42e68ab03f84d4b92893006fcf6f229 from peer 68a3c2b17ed94e06aba66ed4178132ab at {username='slave'} at 127.19.94.199:38699: session id = 68a3c2b17ed94e06aba66ed4178132ab-a42e68ab03f84d4b92893006fcf6f229
I20240502 14:16:53.881790 26839 tablet_copy_source_session.cc:205] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:53.883834 11936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a42e68ab03f84d4b92893006fcf6f229. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:53.897974 11936 tablet_copy_client.cc:783] T a42e68ab03f84d4b92893006fcf6f229 P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:53.898226 11936 tablet_copy_client.cc:647] T a42e68ab03f84d4b92893006fcf6f229 P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:53.899688 11936 tablet_copy_client.cc:523] T a42e68ab03f84d4b92893006fcf6f229 P 68a3c2b17ed94e06aba66ed4178132ab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:53.903935 11936 tablet_bootstrap.cc:492] T a42e68ab03f84d4b92893006fcf6f229 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap starting.
I20240502 14:16:53.916276 29621 maintenance_manager.cc:419] P 4092902b3e674857a0147f1b4f2cdb86: Scheduling UndoDeltaBlockGCOp(bb3e4ec75888419aa4e065cf5513ae5b): 4925 bytes on disk
I20240502 14:16:53.917078 29512 maintenance_manager.cc:643] P 4092902b3e674857a0147f1b4f2cdb86: UndoDeltaBlockGCOp(bb3e4ec75888419aa4e065cf5513ae5b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
I20240502 14:16:53.924119 29621 maintenance_manager.cc:419] P 4092902b3e674857a0147f1b4f2cdb86: Scheduling FlushMRSOp(825ae3e90e214a6f98394765edf6ff2d): perf score=0.033342
I20240502 14:16:53.949280 11936 tablet_bootstrap.cc:492] T a42e68ab03f84d4b92893006fcf6f229 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:53.949687 11936 tablet_bootstrap.cc:492] T a42e68ab03f84d4b92893006fcf6f229 P 68a3c2b17ed94e06aba66ed4178132ab: Bootstrap complete.
I20240502 14:16:53.950130 11936 ts_tablet_manager.cc:1367] T a42e68ab03f84d4b92893006fcf6f229 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent bootstrapping tablet: real 0.046s user 0.023s sys 0.013s
I20240502 14:16:53.952533 11936 raft_consensus.cc:348] T a42e68ab03f84d4b92893006fcf6f229 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:16:53.952781 11936 raft_consensus.cc:729] T a42e68ab03f84d4b92893006fcf6f229 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Initialized, Role: FOLLOWER
I20240502 14:16:53.953058 11936 consensus_queue.cc:260] T a42e68ab03f84d4b92893006fcf6f229 P 68a3c2b17ed94e06aba66ed4178132ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:16:53.953966 11936 ts_tablet_manager.cc:1398] T a42e68ab03f84d4b92893006fcf6f229 P 68a3c2b17ed94e06aba66ed4178132ab: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:16:53.955050 26839 tablet_copy_service.cc:342] P e4cfb8fc5a474177af6848a6efa13a27: Request end of tablet copy session 68a3c2b17ed94e06aba66ed4178132ab-a42e68ab03f84d4b92893006fcf6f229 received from {username='slave'} at 127.19.94.199:38699
I20240502 14:16:53.955180 26839 tablet_copy_service.cc:433] P e4cfb8fc5a474177af6848a6efa13a27: ending tablet copy session 68a3c2b17ed94e06aba66ed4178132ab-a42e68ab03f84d4b92893006fcf6f229 on tablet a42e68ab03f84d4b92893006fcf6f229 with peer 68a3c2b17ed94e06aba66ed4178132ab
I20240502 14:16:54.002269 29575 raft_consensus.cc:1203] T bb3e4ec75888419aa4e065cf5513ae5b P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 14:16:54.008199 26557 raft_consensus.cc:1203] T 82e51aa5371e4731bc00daafc65c7a88 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:16:54.048013 26686 consensus_queue.cc:237] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:16:54.057439 12010 ts_tablet_manager.cc:902] T 310aa6ad3604487589dbe96afa25bb05 P c1100e16b3b9495ca61d4f95e89b6a89: Initiating tablet copy from peer ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:16:54.087103 12010 tablet_copy_client.cc:308] T 310aa6ad3604487589dbe96afa25bb05 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.194:36025
I20240502 14:16:54.087850 26578 tablet_copy_service.cc:140] P ba187e38a2754d60a12de9ac4856430a: Received BeginTabletCopySession request for tablet 310aa6ad3604487589dbe96afa25bb05 from peer c1100e16b3b9495ca61d4f95e89b6a89 ({username='slave'} at 127.19.94.198:36021)
I20240502 14:16:54.088045 26578 tablet_copy_service.cc:161] P ba187e38a2754d60a12de9ac4856430a: Beginning new tablet copy session on tablet 310aa6ad3604487589dbe96afa25bb05 from peer c1100e16b3b9495ca61d4f95e89b6a89 at {username='slave'} at 127.19.94.198:36021: session id = c1100e16b3b9495ca61d4f95e89b6a89-310aa6ad3604487589dbe96afa25bb05
I20240502 14:16:54.091790 26578 tablet_copy_source_session.cc:205] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:16:54.093633 12010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 310aa6ad3604487589dbe96afa25bb05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:54.106662 12010 tablet_copy_client.cc:783] T 310aa6ad3604487589dbe96afa25bb05 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Starting download of 5 data blocks...
I20240502 14:16:54.115470 12010 tablet_copy_client.cc:647] T 310aa6ad3604487589dbe96afa25bb05 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:54.117501 12010 tablet_copy_client.cc:523] T 310aa6ad3604487589dbe96afa25bb05 P c1100e16b3b9495ca61d4f95e89b6a89: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:54.127425 12010 tablet_bootstrap.cc:492] T 310aa6ad3604487589dbe96afa25bb05 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap starting.
W20240502 14:16:54.130223 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): Couldn't send request to peer e4cfb8fc5a474177af6848a6efa13a27. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 451b303a32a3404d8d584203d6704b35. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:54.131455 29438 raft_consensus.cc:1261] T 451b303a32a3404d8d584203d6704b35 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:54.133821 11757 consensus_queue.cc:1035] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:54.144629 29175 raft_consensus.cc:1261] T 451b303a32a3404d8d584203d6704b35 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:54.145340 11988 consensus_queue.cc:1035] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:54.146926 29512 maintenance_manager.cc:643] P 4092902b3e674857a0147f1b4f2cdb86: FlushMRSOp(825ae3e90e214a6f98394765edf6ff2d) complete. Timing: real 0.222s user 0.060s sys 0.016s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":14506,"dirs.queue_time_us":12921,"dirs.run_cpu_time_us":257,"dirs.run_wall_time_us":6437,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":997,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":37935,"threads_started":3,"wal-append.queue_time_us":10323}
I20240502 14:16:54.152987 29621 maintenance_manager.cc:419] P 4092902b3e674857a0147f1b4f2cdb86: Scheduling UndoDeltaBlockGCOp(825ae3e90e214a6f98394765edf6ff2d): 4840 bytes on disk
I20240502 14:16:54.154659 29512 maintenance_manager.cc:643] P 4092902b3e674857a0147f1b4f2cdb86: UndoDeltaBlockGCOp(825ae3e90e214a6f98394765edf6ff2d) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":192,"lbm_reads_lt_1ms":4}
I20240502 14:16:54.191807 12010 tablet_bootstrap.cc:492] T 310aa6ad3604487589dbe96afa25bb05 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:54.193881 12010 tablet_bootstrap.cc:492] T 310aa6ad3604487589dbe96afa25bb05 P c1100e16b3b9495ca61d4f95e89b6a89: Bootstrap complete.
I20240502 14:16:54.194311 12010 ts_tablet_manager.cc:1367] T 310aa6ad3604487589dbe96afa25bb05 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent bootstrapping tablet: real 0.067s user 0.020s sys 0.007s
I20240502 14:16:54.196106 12010 raft_consensus.cc:348] T 310aa6ad3604487589dbe96afa25bb05 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:16:54.196488 12010 raft_consensus.cc:729] T 310aa6ad3604487589dbe96afa25bb05 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1100e16b3b9495ca61d4f95e89b6a89, State: Initialized, Role: FOLLOWER
I20240502 14:16:54.196851 12010 consensus_queue.cc:260] T 310aa6ad3604487589dbe96afa25bb05 P c1100e16b3b9495ca61d4f95e89b6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:16:54.197607 12010 ts_tablet_manager.cc:1398] T 310aa6ad3604487589dbe96afa25bb05 P c1100e16b3b9495ca61d4f95e89b6a89: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:16:54.204788 26578 tablet_copy_service.cc:342] P ba187e38a2754d60a12de9ac4856430a: Request end of tablet copy session c1100e16b3b9495ca61d4f95e89b6a89-310aa6ad3604487589dbe96afa25bb05 received from {username='slave'} at 127.19.94.198:36021
I20240502 14:16:54.204957 26578 tablet_copy_service.cc:433] P ba187e38a2754d60a12de9ac4856430a: ending tablet copy session c1100e16b3b9495ca61d4f95e89b6a89-310aa6ad3604487589dbe96afa25bb05 on tablet 310aa6ad3604487589dbe96afa25bb05 with peer c1100e16b3b9495ca61d4f95e89b6a89
W20240502 14:16:54.302556 29511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 55207.7us
W20240502 14:16:54.304482 29510 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.197:38715, user_credentials={real_user=slave}} blocked reactor thread for 124978us
I20240502 14:16:54.338587 11988 raft_consensus.cc:2932] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:54.339612 29355 maintenance_manager.cc:419] P c1100e16b3b9495ca61d4f95e89b6a89: Scheduling UndoDeltaBlockGCOp(310aa6ad3604487589dbe96afa25bb05): 4586 bytes on disk
I20240502 14:16:54.340360 29246 maintenance_manager.cc:643] P c1100e16b3b9495ca61d4f95e89b6a89: UndoDeltaBlockGCOp(310aa6ad3604487589dbe96afa25bb05) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4}
I20240502 14:16:54.341634 29174 raft_consensus.cc:2932] T 451b303a32a3404d8d584203d6704b35 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:54.353211 29443 raft_consensus.cc:2932] T 451b303a32a3404d8d584203d6704b35 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:54.356595 26280 catalog_manager.cc:5484] T 451b303a32a3404d8d584203d6704b35 P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: config changed from index 3 to 4, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:54.382218 29438 raft_consensus.cc:1203] T a42e68ab03f84d4b92893006fcf6f229 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:16:54.425871 29222 maintenance_manager.cc:419] P 1cb67158133e4648a6eceba32d4dbba3: Scheduling FlushMRSOp(981e25d29abc44299dc115d6df835c58): perf score=0.033383
I20240502 14:16:54.438407 29306 raft_consensus.cc:1203] T 310aa6ad3604487589dbe96afa25bb05 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 14:16:54.508719 12036 ts_tablet_manager.cc:902] T 451b303a32a3404d8d584203d6704b35 P e4cfb8fc5a474177af6848a6efa13a27: Initiating tablet copy from peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:16:54.618907 12036 tablet_copy_client.cc:308] T 451b303a32a3404d8d584203d6704b35 P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.195:44325
I20240502 14:16:54.621258 26709 tablet_copy_service.cc:140] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Received BeginTabletCopySession request for tablet 451b303a32a3404d8d584203d6704b35 from peer e4cfb8fc5a474177af6848a6efa13a27 ({username='slave'} at 127.19.94.196:37053)
I20240502 14:16:54.621443 26709 tablet_copy_service.cc:161] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Beginning new tablet copy session on tablet 451b303a32a3404d8d584203d6704b35 from peer e4cfb8fc5a474177af6848a6efa13a27 at {username='slave'} at 127.19.94.196:37053: session id = e4cfb8fc5a474177af6848a6efa13a27-451b303a32a3404d8d584203d6704b35
I20240502 14:16:54.628455 26709 tablet_copy_source_session.cc:205] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:54.631636 12036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 451b303a32a3404d8d584203d6704b35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:54.653390 12036 tablet_copy_client.cc:783] T 451b303a32a3404d8d584203d6704b35 P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:54.653882 12036 tablet_copy_client.cc:647] T 451b303a32a3404d8d584203d6704b35 P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:54.660301 12036 tablet_copy_client.cc:523] T 451b303a32a3404d8d584203d6704b35 P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:54.669796 26557 consensus_queue.cc:237] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:54.674163 29574 raft_consensus.cc:1261] T 04f3eff43b544f33b020bd74e4331dea P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:54.674379 26686 raft_consensus.cc:1261] T 04f3eff43b544f33b020bd74e4331dea P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:16:54.675211 26493 consensus_peers.cc:485] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a -> Peer 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): Couldn't send request to peer 89af7f423cf44717a08d6b142f0ed49d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 04f3eff43b544f33b020bd74e4331dea. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:54.676322 11516 consensus_queue.cc:1035] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:54.677350 11866 consensus_queue.cc:1035] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:54.693881 12036 tablet_bootstrap.cc:492] T 451b303a32a3404d8d584203d6704b35 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:16:54.706619 11866 raft_consensus.cc:2932] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:54.730129 29574 raft_consensus.cc:2932] T 04f3eff43b544f33b020bd74e4331dea P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:54.745848 26276 catalog_manager.cc:5484] T 04f3eff43b544f33b020bd74e4331dea P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: config changed from index 3 to 4, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:54.746022 26687 raft_consensus.cc:2932] T 04f3eff43b544f33b020bd74e4331dea P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:54.754069 12036 tablet_bootstrap.cc:492] T 451b303a32a3404d8d584203d6704b35 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:54.758356 12036 tablet_bootstrap.cc:492] T 451b303a32a3404d8d584203d6704b35 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap complete.
I20240502 14:16:54.758915 12036 ts_tablet_manager.cc:1367] T 451b303a32a3404d8d584203d6704b35 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.065s user 0.009s sys 0.002s
I20240502 14:16:54.759956 12036 raft_consensus.cc:348] T 451b303a32a3404d8d584203d6704b35 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:16:54.760190 12036 raft_consensus.cc:729] T 451b303a32a3404d8d584203d6704b35 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:16:54.760504 12036 consensus_queue.cc:260] T 451b303a32a3404d8d584203d6704b35 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:16:54.761204 12036 ts_tablet_manager.cc:1398] T 451b303a32a3404d8d584203d6704b35 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:16:54.762053 26709 tablet_copy_service.cc:342] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Request end of tablet copy session e4cfb8fc5a474177af6848a6efa13a27-451b303a32a3404d8d584203d6704b35 received from {username='slave'} at 127.19.94.196:37053
I20240502 14:16:54.762189 26709 tablet_copy_service.cc:433] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: ending tablet copy session e4cfb8fc5a474177af6848a6efa13a27-451b303a32a3404d8d584203d6704b35 on tablet 451b303a32a3404d8d584203d6704b35 with peer e4cfb8fc5a474177af6848a6efa13a27
I20240502 14:16:54.767144 29113 maintenance_manager.cc:643] P 1cb67158133e4648a6eceba32d4dbba3: FlushMRSOp(981e25d29abc44299dc115d6df835c58) complete. Timing: real 0.341s user 0.080s sys 0.019s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":235843,"compiler_manager_pool.run_cpu_time_us":32,"compiler_manager_pool.run_wall_time_us":32,"dirs.queue_time_us":19099,"dirs.run_cpu_time_us":218,"dirs.run_wall_time_us":940,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":976,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":33633,"threads_started":2,"wal-append.queue_time_us":3328}
I20240502 14:16:54.771157 29222 maintenance_manager.cc:419] P 1cb67158133e4648a6eceba32d4dbba3: Scheduling UndoDeltaBlockGCOp(981e25d29abc44299dc115d6df835c58): 4840 bytes on disk
I20240502 14:16:54.778867 29113 maintenance_manager.cc:643] P 1cb67158133e4648a6eceba32d4dbba3: UndoDeltaBlockGCOp(981e25d29abc44299dc115d6df835c58) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":127,"lbm_reads_lt_1ms":4}
I20240502 14:16:54.892287 29309 consensus_queue.cc:237] T abdd0f01d8d5453793c7e9461af10516 P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:16:54.895521 26557 raft_consensus.cc:1261] T abdd0f01d8d5453793c7e9461af10516 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer c1100e16b3b9495ca61d4f95e89b6a89: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:16:54.895906 29245 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P c1100e16b3b9495ca61d4f95e89b6a89 -> Peer e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): Couldn't send request to peer e4cfb8fc5a474177af6848a6efa13a27. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: abdd0f01d8d5453793c7e9461af10516. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:54.896359 11867 consensus_queue.cc:1035] T abdd0f01d8d5453793c7e9461af10516 P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:54.919044 29438 raft_consensus.cc:1261] T abdd0f01d8d5453793c7e9461af10516 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer c1100e16b3b9495ca61d4f95e89b6a89: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:16:54.919735 12075 consensus_queue.cc:1035] T abdd0f01d8d5453793c7e9461af10516 P c1100e16b3b9495ca61d4f95e89b6a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:54.923812 11867 raft_consensus.cc:2932] T abdd0f01d8d5453793c7e9461af10516 P c1100e16b3b9495ca61d4f95e89b6a89 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:54.930588 29438 raft_consensus.cc:2932] T abdd0f01d8d5453793c7e9461af10516 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:54.937868 26557 raft_consensus.cc:2932] T abdd0f01d8d5453793c7e9461af10516 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:54.959408 26276 catalog_manager.cc:5484] T abdd0f01d8d5453793c7e9461af10516 P 68a3c2b17ed94e06aba66ed4178132ab reported cstate change: config changed from index 3 to 4, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) added. New cstate: current_term: 1 leader_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:16:55.067588 26816 raft_consensus.cc:1203] T 451b303a32a3404d8d584203d6704b35 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:16:55.100764 29355 maintenance_manager.cc:419] P c1100e16b3b9495ca61d4f95e89b6a89: Scheduling FlushMRSOp(6b4745b1153b4acd999ac3a5c137ff53): perf score=0.033362
I20240502 14:16:55.231487 26686 consensus_queue.cc:237] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:16:55.234902 29246 maintenance_manager.cc:643] P c1100e16b3b9495ca61d4f95e89b6a89: FlushMRSOp(6b4745b1153b4acd999ac3a5c137ff53) complete. Timing: real 0.134s user 0.063s sys 0.049s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":24798,"dirs.run_cpu_time_us":226,"dirs.run_wall_time_us":3471,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1371,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":40083,"threads_started":2,"wal-append.queue_time_us":10348}
I20240502 14:16:55.238493 29355 maintenance_manager.cc:419] P c1100e16b3b9495ca61d4f95e89b6a89: Scheduling UndoDeltaBlockGCOp(6b4745b1153b4acd999ac3a5c137ff53): 4923 bytes on disk
I20240502 14:16:55.239197 29246 maintenance_manager.cc:643] P c1100e16b3b9495ca61d4f95e89b6a89: UndoDeltaBlockGCOp(6b4745b1153b4acd999ac3a5c137ff53) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4}
I20240502 14:16:55.248975 12097 ts_tablet_manager.cc:902] T 04f3eff43b544f33b020bd74e4331dea P 89af7f423cf44717a08d6b142f0ed49d: Initiating tablet copy from peer ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
W20240502 14:16:55.248978 26625 consensus_peers.cc:485] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097): Couldn't send request to peer 4092902b3e674857a0147f1b4f2cdb86. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1fb1ef75829641a89c4cac539f431d80. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:55.253630 29174 raft_consensus.cc:1261] T 1fb1ef75829641a89c4cac539f431d80 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:55.254591 26424 raft_consensus.cc:1261] T 1fb1ef75829641a89c4cac539f431d80 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:55.255159 11757 consensus_queue.cc:1035] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:55.255450 12098 consensus_queue.cc:1035] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:55.286640 12097 tablet_copy_client.cc:308] T 04f3eff43b544f33b020bd74e4331dea P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.194:36025
I20240502 14:16:55.287333 26578 tablet_copy_service.cc:140] P ba187e38a2754d60a12de9ac4856430a: Received BeginTabletCopySession request for tablet 04f3eff43b544f33b020bd74e4331dea from peer 89af7f423cf44717a08d6b142f0ed49d ({username='slave'} at 127.19.94.193:46185)
I20240502 14:16:55.287508 26578 tablet_copy_service.cc:161] P ba187e38a2754d60a12de9ac4856430a: Beginning new tablet copy session on tablet 04f3eff43b544f33b020bd74e4331dea from peer 89af7f423cf44717a08d6b142f0ed49d at {username='slave'} at 127.19.94.193:46185: session id = 89af7f423cf44717a08d6b142f0ed49d-04f3eff43b544f33b020bd74e4331dea
I20240502 14:16:55.290225 26578 tablet_copy_source_session.cc:205] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:55.291622 12097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04f3eff43b544f33b020bd74e4331dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:55.291738 11757 raft_consensus.cc:2932] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:16:55.294054 26425 raft_consensus.cc:2932] T 1fb1ef75829641a89c4cac539f431d80 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:16:55.298168 12097 tablet_copy_client.cc:783] T 04f3eff43b544f33b020bd74e4331dea P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:55.298385 12097 tablet_copy_client.cc:647] T 04f3eff43b544f33b020bd74e4331dea P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:55.299245 29175 raft_consensus.cc:2932] T 1fb1ef75829641a89c4cac539f431d80 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:16:55.312994 12097 tablet_copy_client.cc:523] T 04f3eff43b544f33b020bd74e4331dea P 89af7f423cf44717a08d6b142f0ed49d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:55.313162 26280 catalog_manager.cc:5484] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: config changed from index 5 to 6, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) added. New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:16:55.316910 12097 tablet_bootstrap.cc:492] T 04f3eff43b544f33b020bd74e4331dea P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap starting.
I20240502 14:16:55.367408 12097 tablet_bootstrap.cc:492] T 04f3eff43b544f33b020bd74e4331dea P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:55.367905 12097 tablet_bootstrap.cc:492] T 04f3eff43b544f33b020bd74e4331dea P 89af7f423cf44717a08d6b142f0ed49d: Bootstrap complete.
I20240502 14:16:55.368283 12097 ts_tablet_manager.cc:1367] T 04f3eff43b544f33b020bd74e4331dea P 89af7f423cf44717a08d6b142f0ed49d: Time spent bootstrapping tablet: real 0.051s user 0.020s sys 0.019s
I20240502 14:16:55.369042 12097 raft_consensus.cc:348] T 04f3eff43b544f33b020bd74e4331dea P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:55.369277 12097 raft_consensus.cc:729] T 04f3eff43b544f33b020bd74e4331dea P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Initialized, Role: FOLLOWER
I20240502 14:16:55.369639 12097 consensus_queue.cc:260] T 04f3eff43b544f33b020bd74e4331dea P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:55.370354 12097 ts_tablet_manager.cc:1398] T 04f3eff43b544f33b020bd74e4331dea P 89af7f423cf44717a08d6b142f0ed49d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:16:55.371816 26578 tablet_copy_service.cc:342] P ba187e38a2754d60a12de9ac4856430a: Request end of tablet copy session 89af7f423cf44717a08d6b142f0ed49d-04f3eff43b544f33b020bd74e4331dea received from {username='slave'} at 127.19.94.193:46185
I20240502 14:16:55.371939 26578 tablet_copy_service.cc:433] P ba187e38a2754d60a12de9ac4856430a: ending tablet copy session 89af7f423cf44717a08d6b142f0ed49d-04f3eff43b544f33b020bd74e4331dea on tablet 04f3eff43b544f33b020bd74e4331dea with peer 89af7f423cf44717a08d6b142f0ed49d
I20240502 14:16:55.415840 26686 consensus_queue.cc:237] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:16:55.418578 29574 raft_consensus.cc:1261] T 0290ebbcf609427cae8fce6637019336 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 14:16:55.419169 26624 consensus_peers.cc:485] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0290ebbcf609427cae8fce6637019336. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:55.419504 26425 raft_consensus.cc:1261] T 0290ebbcf609427cae8fce6637019336 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:16:55.419183 12106 consensus_queue.cc:1035] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240502 14:16:55.420145 11757 consensus_queue.cc:1035] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:16:55.423820 12106 raft_consensus.cc:2932] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:55.424471 26425 raft_consensus.cc:2932] T 0290ebbcf609427cae8fce6637019336 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:55.428516 29574 raft_consensus.cc:2932] T 0290ebbcf609427cae8fce6637019336 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:55.432193 26276 catalog_manager.cc:5484] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: config changed from index 5 to 6, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:55.500813 12134 ts_tablet_manager.cc:902] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27: Initiating tablet copy from peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:16:55.542320 12134 tablet_copy_client.cc:308] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.198:43659
I20240502 14:16:55.543030 29329 tablet_copy_service.cc:140] P c1100e16b3b9495ca61d4f95e89b6a89: Received BeginTabletCopySession request for tablet abdd0f01d8d5453793c7e9461af10516 from peer e4cfb8fc5a474177af6848a6efa13a27 ({username='slave'} at 127.19.94.196:54193)
I20240502 14:16:55.543206 29329 tablet_copy_service.cc:161] P c1100e16b3b9495ca61d4f95e89b6a89: Beginning new tablet copy session on tablet abdd0f01d8d5453793c7e9461af10516 from peer e4cfb8fc5a474177af6848a6efa13a27 at {username='slave'} at 127.19.94.196:54193: session id = e4cfb8fc5a474177af6848a6efa13a27-abdd0f01d8d5453793c7e9461af10516
I20240502 14:16:55.546279 29329 tablet_copy_source_session.cc:205] T abdd0f01d8d5453793c7e9461af10516 P c1100e16b3b9495ca61d4f95e89b6a89: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:16:55.547952 12134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdd0f01d8d5453793c7e9461af10516. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:16:55.555613 12134 tablet_copy_client.cc:783] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Starting download of 0 data blocks...
I20240502 14:16:55.555943 12134 tablet_copy_client.cc:647] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Starting download of 1 WAL segments...
I20240502 14:16:55.557516 12134 tablet_copy_client.cc:523] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:16:55.562667 12134 tablet_bootstrap.cc:492] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap starting.
I20240502 14:16:55.570343 12134 tablet_bootstrap.cc:492] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:55.571100 12134 tablet_bootstrap.cc:492] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27: Bootstrap complete.
I20240502 14:16:55.571357 12134 ts_tablet_manager.cc:1367] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20240502 14:16:55.572234 12134 raft_consensus.cc:348] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:16:55.572877 12134 raft_consensus.cc:729] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Initialized, Role: FOLLOWER
I20240502 14:16:55.573272 12134 consensus_queue.cc:260] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:16:55.573791 12134 ts_tablet_manager.cc:1398] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:16:55.574488 29329 tablet_copy_service.cc:342] P c1100e16b3b9495ca61d4f95e89b6a89: Request end of tablet copy session e4cfb8fc5a474177af6848a6efa13a27-abdd0f01d8d5453793c7e9461af10516 received from {username='slave'} at 127.19.94.196:54193
I20240502 14:16:55.574594 29329 tablet_copy_service.cc:433] P c1100e16b3b9495ca61d4f95e89b6a89: ending tablet copy session e4cfb8fc5a474177af6848a6efa13a27-abdd0f01d8d5453793c7e9461af10516 on tablet abdd0f01d8d5453793c7e9461af10516 with peer e4cfb8fc5a474177af6848a6efa13a27
I20240502 14:16:55.720955 26425 raft_consensus.cc:1203] T 04f3eff43b544f33b020bd74e4331dea P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:16:55.787916 29489 maintenance_manager.cc:419] P 68a3c2b17ed94e06aba66ed4178132ab: Scheduling FlushMRSOp(ce03ee354c54457487ea939301ccd90e): perf score=0.033334
I20240502 14:16:55.920213 12187 ts_tablet_manager.cc:902] T 0290ebbcf609427cae8fce6637019336 P c1100e16b3b9495ca61d4f95e89b6a89: Initiating tablet copy from peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
F20240502 14:16:55.921809 12187 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0290ebbcf609427cae8fce6637019336.queue_time_us, tablet-download-pool-0290ebbcf609427cae8fce6637019336.queue_time_us) (tablet-download-pool-310aa6ad3604487589dbe96afa25bb05.queue_time_us, tablet-download-pool-310aa6ad3604487589dbe96afa25bb05.queue_time_us) (tablet-download-pool-310aa6ad3604487589dbe96afa25bb05.run_cpu_time_us, tablet-download-pool-310aa6ad3604487589dbe96afa25bb05.run_cpu_time_us) (tablet-download-pool-310aa6ad3604487589dbe96afa25bb05.run_wall_time_us, tablet-download-pool-310aa6ad3604487589dbe96afa25bb05.run_wall_time_us) (tablet-download-pool-40aa23be0caa427e89e8fdaf905dc052.queue_time_us, tablet-download-pool-40aa23be0caa427e89e8fdaf905dc052.queue_time_us) (tablet-download-pool-40aa23be0caa427e89e8fdaf905dc052.run_cpu_time_us, tablet-download-pool-40aa23be0caa427e89e8fdaf905dc052.run_cpu_time_us) (tablet-download-pool-40aa23be0caa427e89e8fdaf905dc052.run_wall_time_us, tablet-download-pool-40aa23be0caa427e89e8fdaf905dc052.run_wall_time_us) (tablet-download-pool-4cea6495041642f58e1eb5d26fc10df5.queue_time_us, tablet-download-pool-4cea6495041642f58e1eb5d26fc10df5.queue_time_us) (tablet-download-pool-4cea6495041642f58e1eb5d26fc10df5.run_cpu_time_us, tablet-download-pool-4cea6495041642f58e1eb5d26fc10df5.run_cpu_time_us) (tablet-download-pool-4cea6495041642f58e1eb5d26fc10df5.run_wall_time_us, tablet-download-pool-4cea6495041642f58e1eb5d26fc10df5.run_wall_time_us) (tablet-download-pool-5e66bc4ba1f84b50884d6492edb1e0d2.queue_time_us, tablet-download-pool-5e66bc4ba1f84b50884d6492edb1e0d2.queue_time_us) (tablet-download-pool-5e66bc4ba1f84b50884d6492edb1e0d2.run_cpu_time_us, tablet-download-pool-5e66bc4ba1f84b50884d6492edb1e0d2.run_cpu_time_us) (tablet-download-pool-5e66bc4ba1f84b50884d6492edb1e0d2.run_wall_time_us, tablet-download-pool-5e66bc4ba1f84b50884d6492edb1e0d2.run_wall_time_us) (tablet-download-pool-66554eb8b44145d4b474707b061c9659.queue_time_us, tablet-download-pool-66554eb8b44145d4b474707b061c9659.queue_time_us) (tablet-download-pool-66554eb8b44145d4b474707b061c9659.run_cpu_time_us, tablet-download-pool-66554eb8b44145d4b474707b061c9659.run_cpu_time_us) (tablet-download-pool-66554eb8b44145d4b474707b061c9659.run_wall_time_us, tablet-download-pool-66554eb8b44145d4b474707b061c9659.run_wall_time_us) (tablet-download-pool-6814613d36a74353a99673cf16aa9821.queue_time_us, tablet-download-pool-6814613d36a74353a99673cf16aa9821.queue_time_us) (tablet-download-pool-6814613d36a74353a99673cf16aa9821.run_cpu_time_us, tablet-download-pool-6814613d36a74353a99673cf16aa9821.run_cpu_time_us) (tablet-download-pool-6814613d36a74353a99673cf16aa9821.run_wall_time_us, tablet-download-pool-6814613d36a74353a99673cf16aa9821.run_wall_time_us) (tablet-download-pool-6b4745b1153b4acd999ac3a5c137ff53.queue_time_us, tablet-download-pool-6b4745b1153b4acd999ac3a5c137ff53.queue_time_us) (tablet-download-pool-6b4745b1153b4acd999ac3a5c137ff53.run_cpu_time_us, tablet-download-pool-6b4745b1153b4acd999ac3a5c137ff53.run_cpu_time_us) (tablet-download-pool-6b4745b1153b4acd999ac3a5c137ff53.run_wall_time_us, tablet-download-pool-6b4745b1153b4acd999ac3a5c137ff53.run_wall_time_us) (tablet-download-pool-946239523c0646f1835d7a85b4e7d117.queue_time_us, tablet-download-pool-946239523c0646f1835d7a85b4e7d117.queue_time_us) (tablet-download-pool-946239523c0646f1835d7a85b4e7d117.run_cpu_time_us, tablet-download-pool-946239523c0646f1835d7a85b4e7d117.run_cpu_time_us) (tablet-download-pool-946239523c0646f1835d7a85b4e7d117.run_wall_time_us, tablet-download-pool-946239523c0646f1835d7a85b4e7d117.run_wall_time_us) (tablet-download-pool-b0316714f34d44a5b537cdf20af42ce9.queue_time_us, tablet-download-pool-b0316714f34d44a5b537cdf20af42ce9.queue_time_us) (tablet-download-pool-b0316714f34d44a5b537cdf20af42ce9.run_cpu_time_us, tablet-download-pool-b0316714f34d44a5b537cdf20af42ce9.run_cpu_time_us) (tablet-download-pool-b0316714f34d44a5b537cdf20af42ce9.run_wall_time_us, tablet-download-pool-b0316714f34d44a5b537cdf20af42ce9.run_wall_time_us) (tablet-download-pool-b8430c13474b4850b71020c4a89bea68.queue_time_us, tablet-download-pool-b8430c13474b4850b71020c4a89bea68.queue_time_us) (tablet-download-pool-b8430c13474b4850b71020c4a89bea68.run_cpu_time_us, tablet-download-pool-b8430c13474b4850b71020c4a89bea68.run_cpu_time_us) (tablet-download-pool-b8430c13474b4850b71020c4a89bea68.run_wall_time_us, tablet-download-pool-b8430c13474b4850b71020c4a89bea68.run_wall_time_us) (tablet-download-pool-bb3e4ec75888419aa4e065cf5513ae5b.queue_time_us, tablet-download-pool-bb3e4ec75888419aa4e065cf5513ae5b.queue_time_us) (tablet-download-pool-bb3e4ec75888419aa4e065cf5513ae5b.run_cpu_time_us, tablet-download-pool-bb3e4ec75888419aa4e065cf5513ae5b.run_cpu_time_us) (tablet-download-pool-bb3e4ec75888419aa4e065cf5513ae5b.run_wall_time_us, tablet-download-pool-bb3e4ec75888419aa4e065cf5513ae5b.run_wall_time_us) (tablet-download-pool-c9fb1f7eb71c42d2b8f0f6aacd2b0aeb.queue_time_us, tablet-download-pool-c9fb1f7eb71c42d2b8f0f6aacd2b0aeb.queue_time_us) (tablet-download-pool-c9fb1f7eb71c42d2b8f0f6aacd2b0aeb.run_cpu_time_us, tablet-download-pool-c9fb1f7eb71c42d2b8f0f6aacd2b0aeb.run_cpu_time_us) (tablet-download-pool-c9fb1f7eb71c42d2b8f0f6aacd2b0aeb.run_wall_time_us, tablet-download-pool-c9fb1f7eb71c42d2b8f0f6aacd2b0aeb.run_wall_time_us) (tablet-download-pool-d7f02f897ce54fefb03832da3c2d95e0.queue_time_us, tablet-download-pool-d7f02f897ce54fefb03832da3c2d95e0.queue_time_us) (tablet-download-pool-d7f02f897ce54fefb03832da3c2d95e0.run_cpu_time_us, tablet-download-pool-d7f02f897ce54fefb03832da3c2d95e0.run_cpu_time_us) (tablet-download-pool-d7f02f897ce54fefb03832da3c2d95e0.run_wall_time_us, tablet-download-pool-d7f02f897ce54fefb03832da3c2d95e0.run_wall_time_us) (tablet-download-pool-e1bdebd770244537a0119baa4e3f8091.queue_time_us, tablet-download-pool-e1bdebd770244537a0119baa4e3f8091.queue_time_us) (tablet-download-pool-e1bdebd770244537a0119baa4e3f8091.run_cpu_time_us, tablet-download-pool-e1bdebd770244537a0119baa4e3f8091.run_cpu_time_us) (tablet-download-pool-e1bdebd770244537a0119baa4e3f8091.run_wall_time_us, tablet-download-pool-e1bdebd770244537a0119baa4e3f8091.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240502 14:16:55.931510 12186 ts_tablet_manager.cc:902] T 1fb1ef75829641a89c4cac539f431d80 P 4092902b3e674857a0147f1b4f2cdb86: Initiating tablet copy from peer 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
@ 0x7fe724cf1dcd google::LogMessage::Fail() at ??:0
I20240502 14:16:55.936177 29621 maintenance_manager.cc:419] P 4092902b3e674857a0147f1b4f2cdb86: Scheduling FlushMRSOp(058eec4efd564c348684857372c60127): perf score=0.033401
@ 0x7fe724cf5b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fe724cf17cc google::LogMessage::Flush() at ??:0
@ 0x7fe724cf2f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe727a32742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7fe7279fef8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7fe7279f8cd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7fe736e4b8de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240502 14:16:56.022377 26815 raft_consensus.cc:1203] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
@ 0x7fe736e63daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240502 14:16:56.034507 12186 tablet_copy_client.cc:308] T 1fb1ef75829641a89c4cac539f431d80 P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.195:44325
I20240502 14:16:56.035274 26709 tablet_copy_service.cc:140] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Received BeginTabletCopySession request for tablet 1fb1ef75829641a89c4cac539f431d80 from peer 4092902b3e674857a0147f1b4f2cdb86 ({username='slave'} at 127.19.94.200:47557)
I20240502 14:16:56.035439 26709 tablet_copy_service.cc:161] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Beginning new tablet copy session on tablet 1fb1ef75829641a89c4cac539f431d80 from peer 4092902b3e674857a0147f1b4f2cdb86 at {username='slave'} at 127.19.94.200:47557: session id = 4092902b3e674857a0147f1b4f2cdb86-1fb1ef75829641a89c4cac539f431d80
@ 0x7fe736f8e480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240502 14:16:56.041539 26709 tablet_copy_source_session.cc:205] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:16:56.043855 12186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fb1ef75829641a89c4cac539f431d80. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fe736fc6a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240502 14:16:56.057973 12186 tablet_copy_client.cc:783] T 1fb1ef75829641a89c4cac539f431d80 P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Starting download of 5 data blocks...
@ 0x7fe736fa9981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7fe736fa970e std::_Function_handler<>::_M_invoke() at ??:0
I20240502 14:16:56.072438 12186 tablet_copy_client.cc:647] T 1fb1ef75829641a89c4cac539f431d80 P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Starting download of 1 WAL segments...
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240502 14:16:56.075194 12186 tablet_copy_client.cc:523] T 1fb1ef75829641a89c4cac539f431d80 P 4092902b3e674857a0147f1b4f2cdb86: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x7fe727a02b75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe727a0619c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fe727a06019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240502 14:16:56.093333 12186 tablet_bootstrap.cc:492] T 1fb1ef75829641a89c4cac539f431d80 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap starting.
@ 0x7fe7279e101e kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe72ad666db start_thread at ??:0
@ 0x7fe723ba471f clone at ??:0
I20240502 14:16:56.162112 29155 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:56.188962 29421 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:16:56.220577 29380 maintenance_manager.cc:643] P 68a3c2b17ed94e06aba66ed4178132ab: FlushMRSOp(ce03ee354c54457487ea939301ccd90e) complete. Timing: real 0.432s user 0.068s sys 0.055s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":20680,"dirs.queue_time_us":92200,"dirs.run_cpu_time_us":259,"dirs.run_wall_time_us":1499,"drs_written":1,"lbm_read_time_us":119,"lbm_reads_lt_1ms":4,"lbm_write_time_us":918,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":218521,"threads_started":3,"wal-append.queue_time_us":50727}
I20240502 14:16:56.226913 29489 maintenance_manager.cc:419] P 68a3c2b17ed94e06aba66ed4178132ab: Scheduling UndoDeltaBlockGCOp(ce03ee354c54457487ea939301ccd90e): 4925 bytes on disk
I20240502 14:16:56.229014 29380 maintenance_manager.cc:643] P 68a3c2b17ed94e06aba66ed4178132ab: UndoDeltaBlockGCOp(ce03ee354c54457487ea939301ccd90e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":136,"lbm_reads_lt_1ms":4}
I20240502 14:16:56.240639 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240502 14:16:56.253976 29112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.196:32843, user_credentials={real_user=slave}} blocked reactor thread for 52993.3us
I20240502 14:16:56.269626 12186 tablet_bootstrap.cc:492] T 1fb1ef75829641a89c4cac539f431d80 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:16:56.270112 12186 tablet_bootstrap.cc:492] T 1fb1ef75829641a89c4cac539f431d80 P 4092902b3e674857a0147f1b4f2cdb86: Bootstrap complete.
I20240502 14:16:56.270403 12186 ts_tablet_manager.cc:1367] T 1fb1ef75829641a89c4cac539f431d80 P 4092902b3e674857a0147f1b4f2cdb86: Time spent bootstrapping tablet: real 0.177s user 0.019s sys 0.003s
I20240502 14:16:56.276298 26406 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:56.291324 12186 raft_consensus.cc:348] T 1fb1ef75829641a89c4cac539f431d80 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:16:56.291577 12186 raft_consensus.cc:729] T 1fb1ef75829641a89c4cac539f431d80 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Initialized, Role: FOLLOWER
I20240502 14:16:56.292542 12186 consensus_queue.cc:260] T 1fb1ef75829641a89c4cac539f431d80 P 4092902b3e674857a0147f1b4f2cdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:16:56.303073 12186 ts_tablet_manager.cc:1398] T 1fb1ef75829641a89c4cac539f431d80 P 4092902b3e674857a0147f1b4f2cdb86: Time spent starting tablet: real 0.033s user 0.000s sys 0.003s
I20240502 14:16:56.310750 26538 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:56.315420 26709 tablet_copy_service.cc:342] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Request end of tablet copy session 4092902b3e674857a0147f1b4f2cdb86-1fb1ef75829641a89c4cac539f431d80 received from {username='slave'} at 127.19.94.200:47557
I20240502 14:16:56.315582 26709 tablet_copy_service.cc:433] P 69ab78b2a1b74d2bbf2cdeafa0a4af23: ending tablet copy session 4092902b3e674857a0147f1b4f2cdb86-1fb1ef75829641a89c4cac539f431d80 on tablet 1fb1ef75829641a89c4cac539f431d80 with peer 4092902b3e674857a0147f1b4f2cdb86
I20240502 14:16:56.328670 29554 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:16:56.362372 29574 raft_consensus.cc:1203] T 1fb1ef75829641a89c4cac539f431d80 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 14:16:56.495757 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:16:56.518051 29512 maintenance_manager.cc:643] P 4092902b3e674857a0147f1b4f2cdb86: FlushMRSOp(058eec4efd564c348684857372c60127) complete. Timing: real 0.581s user 0.048s sys 0.013s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":79231,"dirs.run_cpu_time_us":242,"dirs.run_wall_time_us":3277,"drs_written":1,"lbm_read_time_us":120,"lbm_reads_lt_1ms":4,"lbm_write_time_us":11598,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":159421,"threads_started":2,"wal-append.queue_time_us":30315}
I20240502 14:16:56.527406 29621 maintenance_manager.cc:419] P 4092902b3e674857a0147f1b4f2cdb86: Scheduling FlushMRSOp(226b35583ee4441a82a66520aeffbb7d): perf score=0.033366
I20240502 14:16:56.557533 29222 maintenance_manager.cc:419] P 1cb67158133e4648a6eceba32d4dbba3: Scheduling FlushMRSOp(310aa6ad3604487589dbe96afa25bb05): perf score=0.033384
W20240502 14:16:56.740119 26624 connection.cc:535] client connection to 127.19.94.198:43659 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:16:56.740409 26624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:16:56.742051 29512 maintenance_manager.cc:643] P 4092902b3e674857a0147f1b4f2cdb86: FlushMRSOp(226b35583ee4441a82a66520aeffbb7d) complete. Timing: real 0.214s user 0.057s sys 0.010s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":43,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":1918,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":848,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":806,"threads_started":1,"wal-append.queue_time_us":327}
I20240502 14:16:56.744884 29621 maintenance_manager.cc:419] P 4092902b3e674857a0147f1b4f2cdb86: Scheduling UndoDeltaBlockGCOp(058eec4efd564c348684857372c60127): 4840 bytes on disk
W20240502 14:16:56.747679 26754 connection.cc:535] client connection to 127.19.94.198:43659 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:16:56.747519 29113 maintenance_manager.cc:643] P 1cb67158133e4648a6eceba32d4dbba3: FlushMRSOp(310aa6ad3604487589dbe96afa25bb05) complete. Timing: real 0.190s user 0.053s sys 0.011s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":18346,"dirs.run_cpu_time_us":238,"dirs.run_wall_time_us":13028,"drs_written":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1005,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":28050,"threads_started":2,"wal-append.queue_time_us":297}
W20240502 14:16:56.747948 26754 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:16:56.750162 29222 maintenance_manager.cc:419] P 1cb67158133e4648a6eceba32d4dbba3: Scheduling UndoDeltaBlockGCOp(310aa6ad3604487589dbe96afa25bb05): 4586 bytes on disk
I20240502 14:16:56.750857 29512 maintenance_manager.cc:643] P 4092902b3e674857a0147f1b4f2cdb86: UndoDeltaBlockGCOp(058eec4efd564c348684857372c60127) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4}
I20240502 14:16:56.751583 29113 maintenance_manager.cc:643] P 1cb67158133e4648a6eceba32d4dbba3: UndoDeltaBlockGCOp(310aa6ad3604487589dbe96afa25bb05) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240502 14:16:56.757246 29621 maintenance_manager.cc:419] P 4092902b3e674857a0147f1b4f2cdb86: Scheduling UndoDeltaBlockGCOp(226b35583ee4441a82a66520aeffbb7d): 4840 bytes on disk
I20240502 14:16:56.758051 29512 maintenance_manager.cc:643] P 4092902b3e674857a0147f1b4f2cdb86: UndoDeltaBlockGCOp(226b35583ee4441a82a66520aeffbb7d) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":135,"lbm_reads_lt_1ms":4}
W20240502 14:16:56.759513 29111 connection.cc:535] client connection to 127.19.94.198:43659 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:16:56.759795 29111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:16:56.762830 29621 maintenance_manager.cc:419] P 4092902b3e674857a0147f1b4f2cdb86: Scheduling UndoDeltaBlockGCOp(1fb1ef75829641a89c4cac539f431d80): 4586 bytes on disk
I20240502 14:16:56.763567 29512 maintenance_manager.cc:643] P 4092902b3e674857a0147f1b4f2cdb86: UndoDeltaBlockGCOp(1fb1ef75829641a89c4cac539f431d80) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
W20240502 14:16:56.767143 26493 connection.cc:535] client connection to 127.19.94.198:43659 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:16:56.767452 26493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:16:56.768886 29510 connection.cc:535] client connection to 127.19.94.198:43659 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:16:56.769089 26362 connection.cc:535] client connection to 127.19.94.198:43659 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:16:56.769132 29510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:16:56.769320 26362 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:16:56.778759 29378 connection.cc:535] client connection to 127.19.94.198:43659 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:16:56.779032 29378 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:16:56.787920 12238 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:16:56.757367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.94.198:43659 (local address 127.19.94.196:57567)
0502 14:16:56.758248 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0502 14:16:56.758283 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0502 14:16:56.763318 (+ 5035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:16:56.785386 (+ 22068us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":264,"threads_started":1}
W20240502 14:16:56.792485 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:56.792614 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:56.792675 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:56.792732 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:56.795066 12236 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:16:56.742882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.94.198:43659 (local address 127.19.94.195:36367)
0502 14:16:56.794217 (+ 51335us) negotiation.cc:107] Waiting for socket to connect
0502 14:16:56.794400 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":51141,"threads_started":1}
W20240502 14:16:56.796612 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection reset by peer (error 104)
W20240502 14:16:56.796747 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:56.799723 29378 consensus_peers.cc:485] T c32ea761f2834d758529890e5b496f88 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:56.799911 29378 consensus_peers.cc:485] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:56.800161 29378 consensus_peers.cc:485] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:56.800273 29378 consensus_peers.cc:485] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:56.800472 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:56.800575 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:56.800632 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:56.800685 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:56.801867 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:56.801995 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:56.802071 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:56.802140 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:56.800729 29378 consensus_peers.cc:485] T a511c4f2b4c14f4a90872bf0093c035c P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:56.802810 29378 consensus_peers.cc:485] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:56.802917 29378 consensus_peers.cc:485] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:56.800748 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:56.803762 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:56.803853 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:56.803908 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:56.804333 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:56.813215 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:56.823909 26362 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.198:43659, user_credentials={real_user=slave}} blocked reactor thread for 54601.9us
W20240502 14:16:56.833210 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:56.833415 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:56.833508 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:56.833606 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:56.837318 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:16:57.065795 11696 raft_consensus.cc:482] T 56285a2fdcdd47e9b9c81b4b8993c4aa P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c1100e16b3b9495ca61d4f95e89b6a89)
I20240502 14:16:57.066023 11696 raft_consensus.cc:504] T 56285a2fdcdd47e9b9c81b4b8993c4aa P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:57.066737 11696 leader_election.cc:288] T 56285a2fdcdd47e9b9c81b4b8993c4aa P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
W20240502 14:16:57.069305 26362 leader_election.cc:334] T 56285a2fdcdd47e9b9c81b4b8993c4aa P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:16:57.070605 26556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56285a2fdcdd47e9b9c81b4b8993c4aa" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:16:57.078505 26362 leader_election.cc:302] T 56285a2fdcdd47e9b9c81b4b8993c4aa P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d; no voters: ba187e38a2754d60a12de9ac4856430a, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:16:57.079272 12108 raft_consensus.cc:2726] T 56285a2fdcdd47e9b9c81b4b8993c4aa P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:16:57.155180 11696 raft_consensus.cc:482] T 3f6d55d89e55425183da73dbb3f4175f P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c1100e16b3b9495ca61d4f95e89b6a89)
I20240502 14:16:57.155395 11696 raft_consensus.cc:504] T 3f6d55d89e55425183da73dbb3f4175f P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:57.156191 11696 leader_election.cc:288] T 3f6d55d89e55425183da73dbb3f4175f P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:16:57.157083 26556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6d55d89e55425183da73dbb3f4175f" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
W20240502 14:16:57.159098 26362 leader_election.cc:334] T 3f6d55d89e55425183da73dbb3f4175f P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:16:57.160295 26362 leader_election.cc:302] T 3f6d55d89e55425183da73dbb3f4175f P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d; no voters: ba187e38a2754d60a12de9ac4856430a, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:16:57.160828 11696 raft_consensus.cc:2726] T 3f6d55d89e55425183da73dbb3f4175f P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:16:57.187072 12128 raft_consensus.cc:482] T 46aaa6186d254ff7bd493bd85479015d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c1100e16b3b9495ca61d4f95e89b6a89)
I20240502 14:16:57.187280 12128 raft_consensus.cc:504] T 46aaa6186d254ff7bd493bd85479015d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:16:57.187961 12128 leader_election.cc:288] T 46aaa6186d254ff7bd493bd85479015d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
W20240502 14:16:57.193825 26624 leader_election.cc:334] T 46aaa6186d254ff7bd493bd85479015d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:16:57.203516 29175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46aaa6186d254ff7bd493bd85479015d" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:16:57.205487 26624 leader_election.cc:302] T 46aaa6186d254ff7bd493bd85479015d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters: 1cb67158133e4648a6eceba32d4dbba3, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:16:57.206200 12128 raft_consensus.cc:2726] T 46aaa6186d254ff7bd493bd85479015d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:16:57.233820 12128 raft_consensus.cc:482] T d6c8fa23fffe491fbfcec03afdb2052a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c1100e16b3b9495ca61d4f95e89b6a89)
I20240502 14:16:57.234066 12128 raft_consensus.cc:504] T d6c8fa23fffe491fbfcec03afdb2052a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:57.234797 12128 leader_election.cc:288] T d6c8fa23fffe491fbfcec03afdb2052a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:16:57.247156 12302 raft_consensus.cc:482] T f18990f33bf74c31a0faf5802cc1856d P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c1100e16b3b9495ca61d4f95e89b6a89)
I20240502 14:16:57.247339 12302 raft_consensus.cc:504] T f18990f33bf74c31a0faf5802cc1856d P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:16:57.248142 12302 leader_election.cc:288] T f18990f33bf74c31a0faf5802cc1856d P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:16:57.248621 29574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f18990f33bf74c31a0faf5802cc1856d" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86" is_pre_election: true
W20240502 14:16:57.250022 26624 leader_election.cc:334] T d6c8fa23fffe491fbfcec03afdb2052a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:16:57.250692 29378 leader_election.cc:334] T f18990f33bf74c31a0faf5802cc1856d P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:16:57.250825 29378 leader_election.cc:302] T f18990f33bf74c31a0faf5802cc1856d P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab; no voters: 4092902b3e674857a0147f1b4f2cdb86, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:16:57.251771 12302 raft_consensus.cc:2726] T f18990f33bf74c31a0faf5802cc1856d P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:16:57.255177 26424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c8fa23fffe491fbfcec03afdb2052a" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:16:57.258009 26624 leader_election.cc:302] T d6c8fa23fffe491fbfcec03afdb2052a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters: 89af7f423cf44717a08d6b142f0ed49d, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:16:57.258620 29222 maintenance_manager.cc:419] P 1cb67158133e4648a6eceba32d4dbba3: Scheduling FlushMRSOp(d1aa4f4be5894ec5a64275f1172e1f55): perf score=0.033362
I20240502 14:16:57.260957 12299 raft_consensus.cc:2726] T d6c8fa23fffe491fbfcec03afdb2052a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:16:57.282924 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:16:57.285993 11866 raft_consensus.cc:482] T abdd0f01d8d5453793c7e9461af10516 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c1100e16b3b9495ca61d4f95e89b6a89)
I20240502 14:16:57.286262 11866 raft_consensus.cc:504] T abdd0f01d8d5453793c7e9461af10516 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:16:57.287269 11866 leader_election.cc:288] T abdd0f01d8d5453793c7e9461af10516 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:16:57.289201 26815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdd0f01d8d5453793c7e9461af10516" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
W20240502 14:16:57.290787 26493 leader_election.cc:334] T abdd0f01d8d5453793c7e9461af10516 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:16:57.290933 26493 leader_election.cc:302] T abdd0f01d8d5453793c7e9461af10516 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: c1100e16b3b9495ca61d4f95e89b6a89, e4cfb8fc5a474177af6848a6efa13a27
I20240502 14:16:57.291313 11866 raft_consensus.cc:2726] T abdd0f01d8d5453793c7e9461af10516 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:16:57.291869 29438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdd0f01d8d5453793c7e9461af10516" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:16:57.311071 11696 raft_consensus.cc:482] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c1100e16b3b9495ca61d4f95e89b6a89)
I20240502 14:16:57.311337 11696 raft_consensus.cc:504] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:57.312979 11696 leader_election.cc:288] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:16:57.316658 12303 raft_consensus.cc:482] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c1100e16b3b9495ca61d4f95e89b6a89)
I20240502 14:16:57.316831 12303 raft_consensus.cc:504] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:16:57.317618 12303 leader_election.cc:288] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:16:57.318980 26688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46aaa6186d254ff7bd493bd85479015d" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
W20240502 14:16:57.319135 29111 leader_election.cc:334] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:16:57.319248 26688 raft_consensus.cc:2454] T 46aaa6186d254ff7bd493bd85479015d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 1.
I20240502 14:16:57.319991 29112 leader_election.cc:302] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters: c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:16:57.320459 12304 raft_consensus.cc:2781] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:16:57.320546 12304 raft_consensus.cc:482] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c1100e16b3b9495ca61d4f95e89b6a89)
I20240502 14:16:57.320616 12304 raft_consensus.cc:3037] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:57.323379 12304 raft_consensus.cc:504] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
W20240502 14:16:57.325413 29111 leader_election.cc:334] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:16:57.325743 12304 leader_election.cc:288] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 2 election: Requested vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:16:57.325945 26686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c8fa23fffe491fbfcec03afdb2052a" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:16:57.326167 26686 raft_consensus.cc:2454] T d6c8fa23fffe491fbfcec03afdb2052a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:16:57.326654 26686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46aaa6186d254ff7bd493bd85479015d" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:16:57.326926 26686 raft_consensus.cc:3037] T 46aaa6186d254ff7bd493bd85479015d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:57.327071 26363 leader_election.cc:302] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:16:57.327756 12315 raft_consensus.cc:2781] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:16:57.327853 12315 raft_consensus.cc:482] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting leader election (detected failure of leader c1100e16b3b9495ca61d4f95e89b6a89)
I20240502 14:16:57.327953 12315 raft_consensus.cc:3037] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:57.330973 12315 raft_consensus.cc:504] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:57.331923 12315 leader_election.cc:288] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 election: Requested vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325)
I20240502 14:16:57.333034 11913 raft_consensus.cc:482] T 33a3639c26bf4b35a6e15d6ef173751c P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c1100e16b3b9495ca61d4f95e89b6a89)
I20240502 14:16:57.333272 11913 raft_consensus.cc:504] T 33a3639c26bf4b35a6e15d6ef173751c P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:16:57.334355 11913 leader_election.cc:288] T 33a3639c26bf4b35a6e15d6ef173751c P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:16:57.337762 26556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a3639c26bf4b35a6e15d6ef173751c" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
W20240502 14:16:57.340042 29510 leader_election.cc:334] T 33a3639c26bf4b35a6e15d6ef173751c P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:16:57.341313 29510 leader_election.cc:302] T 33a3639c26bf4b35a6e15d6ef173751c P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86; no voters: ba187e38a2754d60a12de9ac4856430a, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:16:57.343781 26688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c8fa23fffe491fbfcec03afdb2052a" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:16:57.344111 26688 raft_consensus.cc:3037] T d6c8fa23fffe491fbfcec03afdb2052a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:57.344632 11913 raft_consensus.cc:2726] T 33a3639c26bf4b35a6e15d6ef173751c P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:16:57.347215 26686 raft_consensus.cc:2454] T 46aaa6186d254ff7bd493bd85479015d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 2.
I20240502 14:16:57.354115 29112 leader_election.cc:302] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters: c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:16:57.354516 12304 raft_consensus.cc:2781] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:16:57.354640 12304 raft_consensus.cc:686] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 [term 2 LEADER]: Becoming Leader. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Running, Role: LEADER
I20240502 14:16:57.354892 12304 consensus_queue.cc:237] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
W20240502 14:16:57.367815 26362 leader_election.cc:334] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:16:57.368211 26362 leader_election.cc:334] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:16:57.369475 26688 raft_consensus.cc:2454] T d6c8fa23fffe491fbfcec03afdb2052a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 2.
I20240502 14:16:57.372704 26285 catalog_manager.cc:5484] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: term changed from 1 to 2, leader changed from c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) to 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197). New cstate: current_term: 2 leader_uuid: "1cb67158133e4648a6eceba32d4dbba3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:16:57.401883 26363 leader_election.cc:302] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, 89af7f423cf44717a08d6b142f0ed49d; no voters: c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:16:57.402658 12108 raft_consensus.cc:2781] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:16:57.402819 12108 raft_consensus.cc:686] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d [term 2 LEADER]: Becoming Leader. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Running, Role: LEADER
I20240502 14:16:57.403098 12108 consensus_queue.cc:237] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:57.409221 12315 raft_consensus.cc:482] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c1100e16b3b9495ca61d4f95e89b6a89)
I20240502 14:16:57.409508 12315 raft_consensus.cc:504] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:57.410698 12315 leader_election.cc:288] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:16:57.413213 26556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "842d4e7e5ee74e309fb7eba6d7b7ae8b" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:16:57.416186 12302 raft_consensus.cc:482] T abdd0f01d8d5453793c7e9461af10516 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c1100e16b3b9495ca61d4f95e89b6a89)
I20240502 14:16:57.416358 12302 raft_consensus.cc:504] T abdd0f01d8d5453793c7e9461af10516 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:16:57.417277 12302 leader_election.cc:288] T abdd0f01d8d5453793c7e9461af10516 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
W20240502 14:16:57.418437 29378 leader_election.cc:334] T abdd0f01d8d5453793c7e9461af10516 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:16:57.419085 26556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdd0f01d8d5453793c7e9461af10516" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:16:57.419369 26556 raft_consensus.cc:2454] T abdd0f01d8d5453793c7e9461af10516 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 1.
I20240502 14:16:57.420534 26815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdd0f01d8d5453793c7e9461af10516" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
W20240502 14:16:57.421193 26362 leader_election.cc:334] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:16:57.421236 29379 leader_election.cc:302] T abdd0f01d8d5453793c7e9461af10516 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, ba187e38a2754d60a12de9ac4856430a; no voters: c1100e16b3b9495ca61d4f95e89b6a89, e4cfb8fc5a474177af6848a6efa13a27
I20240502 14:16:57.421381 26362 leader_election.cc:302] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d; no voters: ba187e38a2754d60a12de9ac4856430a, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:16:57.421720 12329 raft_consensus.cc:2726] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:16:57.422451 12302 raft_consensus.cc:2726] T abdd0f01d8d5453793c7e9461af10516 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:16:57.433107 11866 raft_consensus.cc:482] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c1100e16b3b9495ca61d4f95e89b6a89)
I20240502 14:16:57.433297 11866 raft_consensus.cc:504] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:16:57.434056 11866 leader_election.cc:288] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:16:57.434790 29574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a3639c26bf4b35a6e15d6ef173751c" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86" is_pre_election: true
I20240502 14:16:57.435047 29574 raft_consensus.cc:2454] T 33a3639c26bf4b35a6e15d6ef173751c P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:16:57.435583 26494 leader_election.cc:302] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:16:57.436208 12174 raft_consensus.cc:2781] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:16:57.436324 12174 raft_consensus.cc:482] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting leader election (detected failure of leader c1100e16b3b9495ca61d4f95e89b6a89)
I20240502 14:16:57.436467 12174 raft_consensus.cc:3037] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:57.440785 11866 raft_consensus.cc:482] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c1100e16b3b9495ca61d4f95e89b6a89)
I20240502 14:16:57.440997 11866 raft_consensus.cc:504] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:57.441651 11866 leader_election.cc:288] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:16:57.441793 11866 raft_consensus.cc:482] T 13136deb9046440f88bf3471ff42312c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c1100e16b3b9495ca61d4f95e89b6a89)
I20240502 14:16:57.441934 11866 raft_consensus.cc:504] T 13136deb9046440f88bf3471ff42312c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:57.442495 11866 leader_election.cc:288] T 13136deb9046440f88bf3471ff42312c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:16:57.445683 12174 raft_consensus.cc:504] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:16:57.454579 12174 leader_election.cc:288] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 election: Requested vote from peers 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:16:57.465835 29574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a3639c26bf4b35a6e15d6ef173751c" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86"
I20240502 14:16:57.466097 29574 raft_consensus.cc:3037] T 33a3639c26bf4b35a6e15d6ef173751c P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:57.477082 12307 raft_consensus.cc:482] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c1100e16b3b9495ca61d4f95e89b6a89)
I20240502 14:16:57.477384 12307 raft_consensus.cc:504] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:57.478375 12307 leader_election.cc:288] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:16:57.480355 11696 raft_consensus.cc:482] T 2e59791e95cc452ea30d1ed8bb6178ae P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c1100e16b3b9495ca61d4f95e89b6a89)
I20240502 14:16:57.480556 11696 raft_consensus.cc:504] T 2e59791e95cc452ea30d1ed8bb6178ae P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:57.481210 11696 leader_election.cc:288] T 2e59791e95cc452ea30d1ed8bb6178ae P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:16:57.487004 26280 catalog_manager.cc:5484] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: term changed from 1 to 2, leader changed from c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) to 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193). New cstate: current_term: 2 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:57.489197 29574 raft_consensus.cc:2454] T 33a3639c26bf4b35a6e15d6ef173751c P 4092902b3e674857a0147f1b4f2cdb86 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 2.
W20240502 14:16:57.490271 26362 leader_election.cc:334] T 2e59791e95cc452ea30d1ed8bb6178ae P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:16:57.492947 11913 raft_consensus.cc:482] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c1100e16b3b9495ca61d4f95e89b6a89)
I20240502 14:16:57.493098 11913 raft_consensus.cc:504] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
W20240502 14:16:57.495471 29510 leader_election.cc:334] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:16:57.497210 11913 leader_election.cc:288] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:16:57.498685 26424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e59791e95cc452ea30d1ed8bb6178ae" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:16:57.498970 26424 raft_consensus.cc:2454] T 2e59791e95cc452ea30d1ed8bb6178ae P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
W20240502 14:16:57.455026 26493 leader_election.cc:334] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:16:57.501789 26493 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.94.198:43659, user_credentials={real_user=slave}} blocked reactor thread for 46743.9us
I20240502 14:16:57.482219 12330 raft_consensus.cc:482] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c1100e16b3b9495ca61d4f95e89b6a89)
W20240502 14:16:57.502022 26493 leader_election.cc:334] T 13136deb9046440f88bf3471ff42312c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:16:57.502189 26493 leader_election.cc:334] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:16:57.502100 12330 raft_consensus.cc:504] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:57.502836 12330 leader_election.cc:288] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:16:57.489826 26494 leader_election.cc:302] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:16:57.503647 12330 raft_consensus.cc:2781] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:16:57.503808 12330 raft_consensus.cc:686] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a [term 2 LEADER]: Becoming Leader. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Running, Role: LEADER
I20240502 14:16:57.504091 12330 consensus_queue.cc:237] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:16:57.505359 26624 leader_election.cc:302] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:16:57.512426 12307 raft_consensus.cc:2781] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:16:57.512596 12307 raft_consensus.cc:482] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c1100e16b3b9495ca61d4f95e89b6a89)
I20240502 14:16:57.512703 12307 raft_consensus.cc:3037] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:57.515730 12307 raft_consensus.cc:504] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:57.523509 12015 raft_consensus.cc:482] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c1100e16b3b9495ca61d4f95e89b6a89)
I20240502 14:16:57.523730 12015 raft_consensus.cc:504] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:16:57.524674 12015 leader_election.cc:288] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:16:57.525684 11696 raft_consensus.cc:482] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c1100e16b3b9495ca61d4f95e89b6a89)
I20240502 14:16:57.525897 11696 raft_consensus.cc:504] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:57.526624 11696 leader_election.cc:288] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:16:57.529703 26754 leader_election.cc:334] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:16:57.533597 29113 maintenance_manager.cc:643] P 1cb67158133e4648a6eceba32d4dbba3: FlushMRSOp(d1aa4f4be5894ec5a64275f1172e1f55) complete. Timing: real 0.275s user 0.076s sys 0.064s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":150227,"dirs.run_cpu_time_us":265,"dirs.run_wall_time_us":34887,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":886,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":174346,"threads_started":2,"wal-append.queue_time_us":25076}
I20240502 14:16:57.535717 11696 raft_consensus.cc:482] T 924ce5625bf34a739169b005ee4bedc2 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c1100e16b3b9495ca61d4f95e89b6a89)
I20240502 14:16:57.535926 11696 raft_consensus.cc:504] T 924ce5625bf34a739169b005ee4bedc2 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:57.536607 11696 leader_election.cc:288] T 924ce5625bf34a739169b005ee4bedc2 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:16:57.536651 29222 maintenance_manager.cc:419] P 1cb67158133e4648a6eceba32d4dbba3: Scheduling UndoDeltaBlockGCOp(d1aa4f4be5894ec5a64275f1172e1f55): 4925 bytes on disk
I20240502 14:16:57.537259 29113 maintenance_manager.cc:643] P 1cb67158133e4648a6eceba32d4dbba3: UndoDeltaBlockGCOp(d1aa4f4be5894ec5a64275f1172e1f55) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
I20240502 14:16:57.539753 12302 raft_consensus.cc:482] T cf808a060536445495a70de638438f87 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c1100e16b3b9495ca61d4f95e89b6a89)
I20240502 14:16:57.539927 12302 raft_consensus.cc:504] T cf808a060536445495a70de638438f87 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:57.540771 12302 leader_election.cc:288] T cf808a060536445495a70de638438f87 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
W20240502 14:16:57.542148 29378 leader_election.cc:334] T cf808a060536445495a70de638438f87 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:16:57.543323 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f18990f33bf74c31a0faf5802cc1856d" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:16:57.543563 29443 raft_consensus.cc:2454] T f18990f33bf74c31a0faf5802cc1856d P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 1.
I20240502 14:16:57.543493 12307 leader_election.cc:288] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 election: Requested vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:16:57.544647 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdd0f01d8d5453793c7e9461af10516" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:16:57.545295 29443 raft_consensus.cc:2454] T abdd0f01d8d5453793c7e9461af10516 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
I20240502 14:16:57.553050 29511 leader_election.cc:302] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, 68a3c2b17ed94e06aba66ed4178132ab; no voters: c1100e16b3b9495ca61d4f95e89b6a89
W20240502 14:16:57.553903 26624 leader_election.cc:334] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:16:57.554883 11913 raft_consensus.cc:2781] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240502 14:16:57.554934 26362 leader_election.cc:334] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:16:57.555012 11913 raft_consensus.cc:482] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c1100e16b3b9495ca61d4f95e89b6a89)
I20240502 14:16:57.555114 11913 raft_consensus.cc:3037] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:57.556087 26425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e59791e95cc452ea30d1ed8bb6178ae" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:16:57.556329 26425 raft_consensus.cc:3037] T 2e59791e95cc452ea30d1ed8bb6178ae P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:57.559705 26425 raft_consensus.cc:2454] T 2e59791e95cc452ea30d1ed8bb6178ae P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 2.
I20240502 14:16:57.560002 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf808a060536445495a70de638438f87" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:16:57.560232 26426 raft_consensus.cc:2454] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 1.
I20240502 14:16:57.560705 26688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e59791e95cc452ea30d1ed8bb6178ae" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:16:57.561007 26688 raft_consensus.cc:2379] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 89af7f423cf44717a08d6b142f0ed49d in current term 2: Already voted for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in this term.
W20240502 14:16:57.561471 26362 leader_election.cc:334] T 924ce5625bf34a739169b005ee4bedc2 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:16:57.562268 26688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "924ce5625bf34a739169b005ee4bedc2" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
I20240502 14:16:57.564009 11696 raft_consensus.cc:482] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c1100e16b3b9495ca61d4f95e89b6a89)
I20240502 14:16:57.564105 29378 leader_election.cc:302] T cf808a060536445495a70de638438f87 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, 89af7f423cf44717a08d6b142f0ed49d; no voters: c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:16:57.564234 11696 raft_consensus.cc:504] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:57.564632 12302 raft_consensus.cc:2781] T cf808a060536445495a70de638438f87 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:16:57.564745 12302 raft_consensus.cc:482] T cf808a060536445495a70de638438f87 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Starting leader election (detected failure of leader c1100e16b3b9495ca61d4f95e89b6a89)
I20240502 14:16:57.564854 12302 raft_consensus.cc:3037] T cf808a060536445495a70de638438f87 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:16:57.565141 26493 leader_election.cc:334] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:16:57.565332 26493 leader_election.cc:334] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:16:57.565539 11696 leader_election.cc:288] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:16:57.566238 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6d55d89e55425183da73dbb3f4175f" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:16:57.566483 26426 raft_consensus.cc:2454] T 3f6d55d89e55425183da73dbb3f4175f P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:16:57.567054 26425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13136deb9046440f88bf3471ff42312c" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:16:57.567308 26425 raft_consensus.cc:2454] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:16:57.567654 12302 raft_consensus.cc:504] T cf808a060536445495a70de638438f87 P 68a3c2b17ed94e06aba66ed4178132ab [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:57.567847 26556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13136deb9046440f88bf3471ff42312c" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:16:57.568089 26556 raft_consensus.cc:2454] T 13136deb9046440f88bf3471ff42312c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:16:57.568190 12299 raft_consensus.cc:482] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c1100e16b3b9495ca61d4f95e89b6a89)
I20240502 14:16:57.568529 12299 raft_consensus.cc:504] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:57.569005 26363 leader_election.cc:302] T 2e59791e95cc452ea30d1ed8bb6178ae P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d; no voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:16:57.569247 12299 leader_election.cc:288] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:16:57.569754 26362 leader_election.cc:302] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:16:57.569839 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56285a2fdcdd47e9b9c81b4b8993c4aa" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
W20240502 14:16:57.569994 29378 leader_election.cc:334] T cf808a060536445495a70de638438f87 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:16:57.570021 11696 raft_consensus.cc:2726] T 2e59791e95cc452ea30d1ed8bb6178ae P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:16:57.570027 26426 raft_consensus.cc:2454] T 56285a2fdcdd47e9b9c81b4b8993c4aa P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:16:57.570241 11696 raft_consensus.cc:2781] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:16:57.570323 11696 raft_consensus.cc:482] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting leader election (detected failure of leader c1100e16b3b9495ca61d4f95e89b6a89)
I20240502 14:16:57.570418 11696 raft_consensus.cc:3037] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:57.571159 12302 leader_election.cc:288] T cf808a060536445495a70de638438f87 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 2 election: Requested vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:16:57.573508 29438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf808a060536445495a70de638438f87" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab" is_pre_election: true
I20240502 14:16:57.573585 11696 raft_consensus.cc:504] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:57.573892 29438 raft_consensus.cc:2379] T cf808a060536445495a70de638438f87 P 68a3c2b17ed94e06aba66ed4178132ab [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 89af7f423cf44717a08d6b142f0ed49d in current term 2: Already voted for candidate 68a3c2b17ed94e06aba66ed4178132ab in this term.
I20240502 14:16:57.574287 11696 leader_election.cc:288] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 election: Requested vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:16:57.579416 26424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf808a060536445495a70de638438f87" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:16:57.579433 26556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13136deb9046440f88bf3471ff42312c" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:16:57.579672 26556 raft_consensus.cc:3037] T 13136deb9046440f88bf3471ff42312c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:57.579677 26424 raft_consensus.cc:3037] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:57.580149 26363 leader_election.cc:302] T 924ce5625bf34a739169b005ee4bedc2 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d; no voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:16:57.580669 12108 raft_consensus.cc:2726] T 924ce5625bf34a739169b005ee4bedc2 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:16:57.582801 26556 raft_consensus.cc:2454] T 13136deb9046440f88bf3471ff42312c P ba187e38a2754d60a12de9ac4856430a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 2.
I20240502 14:16:57.583019 26424 raft_consensus.cc:2454] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 2.
W20240502 14:16:57.579680 26362 leader_election.cc:334] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:16:57.587513 29378 leader_election.cc:302] T cf808a060536445495a70de638438f87 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, 89af7f423cf44717a08d6b142f0ed49d; no voters: c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:16:57.588024 12302 raft_consensus.cc:2781] T cf808a060536445495a70de638438f87 P 68a3c2b17ed94e06aba66ed4178132ab [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:16:57.588193 12302 raft_consensus.cc:686] T cf808a060536445495a70de638438f87 P 68a3c2b17ed94e06aba66ed4178132ab [term 2 LEADER]: Becoming Leader. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Running, Role: LEADER
I20240502 14:16:57.588565 12302 consensus_queue.cc:237] T cf808a060536445495a70de638438f87 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:57.590909 12330 raft_consensus.cc:482] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c1100e16b3b9495ca61d4f95e89b6a89)
I20240502 14:16:57.591094 12330 raft_consensus.cc:504] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:57.591786 12330 leader_election.cc:288] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:16:57.591590 26279 catalog_manager.cc:5484] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a reported cstate change: term changed from 1 to 2, leader changed from c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) to ba187e38a2754d60a12de9ac4856430a (127.19.94.194). New cstate: current_term: 2 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
W20240502 14:16:57.596120 29510 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.193:37985, user_credentials={real_user=slave}} blocked reactor thread for 57949.3us
I20240502 14:16:57.600365 26362 leader_election.cc:302] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:16:57.600955 11696 raft_consensus.cc:2781] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:16:57.601249 11696 raft_consensus.cc:686] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d [term 2 LEADER]: Becoming Leader. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Running, Role: LEADER
I20240502 14:16:57.601603 11696 consensus_queue.cc:237] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
W20240502 14:16:57.610493 26362 leader_election.cc:334] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:16:57.602564 26280 catalog_manager.cc:5484] T cf808a060536445495a70de638438f87 P 68a3c2b17ed94e06aba66ed4178132ab reported cstate change: term changed from 1 to 2, leader changed from c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) to 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199). New cstate: current_term: 2 leader_uuid: "68a3c2b17ed94e06aba66ed4178132ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:57.622100 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdd0f01d8d5453793c7e9461af10516" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:16:57.622426 26558 raft_consensus.cc:2454] T abdd0f01d8d5453793c7e9461af10516 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
I20240502 14:16:57.623556 26624 leader_election.cc:302] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, 89af7f423cf44717a08d6b142f0ed49d; no voters:
W20240502 14:16:57.614524 29511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.199:41781, user_credentials={real_user=slave}} blocked reactor thread for 43600.3us
I20240502 14:16:57.624873 26363 leader_election.cc:302] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:16:57.625205 12108 raft_consensus.cc:2726] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:16:57.625963 26754 leader_election.cc:302] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, ba187e38a2754d60a12de9ac4856430a, e4cfb8fc5a474177af6848a6efa13a27; no voters: c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:16:57.626608 11793 raft_consensus.cc:2781] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:16:57.626742 11793 raft_consensus.cc:482] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c1100e16b3b9495ca61d4f95e89b6a89)
I20240502 14:16:57.626889 11793 raft_consensus.cc:3037] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:57.631474 11793 raft_consensus.cc:504] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:16:57.633029 26493 leader_election.cc:302] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, ba187e38a2754d60a12de9ac4856430a; no voters: c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:16:57.633139 26424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "924ce5625bf34a739169b005ee4bedc2" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:16:57.633366 26424 raft_consensus.cc:2454] T 924ce5625bf34a739169b005ee4bedc2 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 1.
I20240502 14:16:57.634100 12334 raft_consensus.cc:2781] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:16:57.634413 26425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "842d4e7e5ee74e309fb7eba6d7b7ae8b" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:16:57.634627 12307 raft_consensus.cc:2781] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:16:57.634817 12307 raft_consensus.cc:686] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 2 LEADER]: Becoming Leader. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Running, Role: LEADER
I20240502 14:16:57.634811 26425 raft_consensus.cc:2454] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
I20240502 14:16:57.635128 12307 consensus_queue.cc:237] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
W20240502 14:16:57.636926 26624 leader_election.cc:334] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:16:57.637606 26754 leader_election.cc:334] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:16:57.639660 11913 raft_consensus.cc:504] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:16:57.640336 11913 leader_election.cc:288] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 2 election: Requested vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:16:57.634215 12334 raft_consensus.cc:482] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting leader election (detected failure of leader c1100e16b3b9495ca61d4f95e89b6a89)
I20240502 14:16:57.640884 12334 raft_consensus.cc:3037] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:57.640936 29438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdd0f01d8d5453793c7e9461af10516" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
I20240502 14:16:57.641336 29438 raft_consensus.cc:3037] T abdd0f01d8d5453793c7e9461af10516 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:57.641076 26624 leader_election.cc:302] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:16:57.633529 26493 leader_election.cc:302] T 13136deb9046440f88bf3471ff42312c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, ba187e38a2754d60a12de9ac4856430a; no voters: c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:16:57.645418 12307 raft_consensus.cc:2781] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:16:57.645561 12307 raft_consensus.cc:482] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c1100e16b3b9495ca61d4f95e89b6a89)
I20240502 14:16:57.645680 12307 raft_consensus.cc:3037] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:57.645788 26493 leader_election.cc:302] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, ba187e38a2754d60a12de9ac4856430a; no voters: c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:16:57.646100 12330 raft_consensus.cc:2741] T 13136deb9046440f88bf3471ff42312c P ba187e38a2754d60a12de9ac4856430a [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20240502 14:16:57.646363 26493 leader_election.cc:302] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:16:57.646507 12330 raft_consensus.cc:2781] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:16:57.647076 12330 raft_consensus.cc:482] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting leader election (detected failure of leader c1100e16b3b9495ca61d4f95e89b6a89)
I20240502 14:16:57.647259 12330 raft_consensus.cc:3037] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:57.647261 26279 catalog_manager.cc:5484] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: term changed from 1 to 2, leader changed from c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) to 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195). New cstate: current_term: 2 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:57.649379 12307 raft_consensus.cc:504] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:57.650197 12307 leader_election.cc:288] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 election: Requested vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:16:57.653988 12334 raft_consensus.cc:504] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:57.654747 12334 leader_election.cc:288] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 election: Requested vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
W20240502 14:16:57.655357 26624 leader_election.cc:334] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:16:57.660234 29510 leader_election.cc:334] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:16:57.663869 26624 leader_election.cc:334] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:16:57.664194 26280 catalog_manager.cc:5484] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: term changed from 1 to 2, leader changed from c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) to 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193). New cstate: current_term: 2 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:57.675673 29438 raft_consensus.cc:2454] T abdd0f01d8d5453793c7e9461af10516 P 68a3c2b17ed94e06aba66ed4178132ab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 2.
I20240502 14:16:57.676249 11793 leader_election.cc:288] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 election: Requested vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:16:57.677883 26425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "924ce5625bf34a739169b005ee4bedc2" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:16:57.678107 26425 raft_consensus.cc:3037] T 924ce5625bf34a739169b005ee4bedc2 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:57.681700 26425 raft_consensus.cc:2454] T 924ce5625bf34a739169b005ee4bedc2 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 2.
I20240502 14:16:57.682559 26624 leader_election.cc:302] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23, 89af7f423cf44717a08d6b142f0ed49d; no voters: c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:16:57.683387 12128 raft_consensus.cc:2781] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:16:57.683532 12128 raft_consensus.cc:686] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 2 LEADER]: Becoming Leader. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Running, Role: LEADER
I20240502 14:16:57.683830 12128 consensus_queue.cc:237] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:57.689399 29438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f18990f33bf74c31a0faf5802cc1856d" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
I20240502 14:16:57.689775 29438 raft_consensus.cc:3037] T f18990f33bf74c31a0faf5802cc1856d P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:57.689507 26280 catalog_manager.cc:5484] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: term changed from 1 to 2, leader changed from c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) to 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195). New cstate: current_term: 2 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:57.669641 12330 raft_consensus.cc:504] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:57.690855 12330 leader_election.cc:288] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 election: Requested vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:16:57.694703 26556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdd0f01d8d5453793c7e9461af10516" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:16:57.695006 26556 raft_consensus.cc:3037] T abdd0f01d8d5453793c7e9461af10516 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:16:57.695566 26493 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.94.193:37985, user_credentials={real_user=slave}} blocked reactor thread for 49242.4us
I20240502 14:16:57.696535 12330 raft_consensus.cc:2781] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:16:57.696645 12330 raft_consensus.cc:482] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting leader election (detected failure of leader c1100e16b3b9495ca61d4f95e89b6a89)
I20240502 14:16:57.696745 12330 raft_consensus.cc:3037] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:57.701932 26426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56285a2fdcdd47e9b9c81b4b8993c4aa" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:16:57.702193 26426 raft_consensus.cc:3037] T 56285a2fdcdd47e9b9c81b4b8993c4aa P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:57.705366 26426 raft_consensus.cc:2454] T 56285a2fdcdd47e9b9c81b4b8993c4aa P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 2.
I20240502 14:16:57.706681 26425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6d55d89e55425183da73dbb3f4175f" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:16:57.707124 26425 raft_consensus.cc:3037] T 3f6d55d89e55425183da73dbb3f4175f P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:57.707197 12330 raft_consensus.cc:504] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
W20240502 14:16:57.706306 26493 leader_election.cc:334] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:16:57.711360 26493 leader_election.cc:302] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:16:57.712528 12330 leader_election.cc:288] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 election: Requested vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:16:57.712796 12330 raft_consensus.cc:2781] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:16:57.713475 12330 raft_consensus.cc:686] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a [term 2 LEADER]: Becoming Leader. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Running, Role: LEADER
I20240502 14:16:57.713989 12330 consensus_queue.cc:237] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:57.714597 26425 raft_consensus.cc:2454] T 3f6d55d89e55425183da73dbb3f4175f P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 2.
W20240502 14:16:57.712811 26493 leader_election.cc:334] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:16:57.715871 26493 leader_election.cc:334] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:16:57.716599 26556 raft_consensus.cc:2454] T abdd0f01d8d5453793c7e9461af10516 P ba187e38a2754d60a12de9ac4856430a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 2.
I20240502 14:16:57.717981 26754 leader_election.cc:302] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 68a3c2b17ed94e06aba66ed4178132ab, ba187e38a2754d60a12de9ac4856430a, e4cfb8fc5a474177af6848a6efa13a27; no voters: c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:16:57.719005 26424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "842d4e7e5ee74e309fb7eba6d7b7ae8b" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:16:57.719278 26424 raft_consensus.cc:3037] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:57.719259 12349 raft_consensus.cc:2781] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:16:57.719501 12349 raft_consensus.cc:686] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 LEADER]: Becoming Leader. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Running, Role: LEADER
I20240502 14:16:57.719915 12349 consensus_queue.cc:237] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:16:57.723593 26424 raft_consensus.cc:2454] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 2.
I20240502 14:16:57.727406 26493 leader_election.cc:302] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, ba187e38a2754d60a12de9ac4856430a; no voters: c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:16:57.727946 26493 leader_election.cc:302] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:16:57.728402 11866 raft_consensus.cc:2781] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:16:57.728461 12334 raft_consensus.cc:2781] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:16:57.728564 11866 raft_consensus.cc:686] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a [term 2 LEADER]: Becoming Leader. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Running, Role: LEADER
I20240502 14:16:57.728637 12334 raft_consensus.cc:686] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a [term 2 LEADER]: Becoming Leader. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Running, Role: LEADER
I20240502 14:16:57.728821 11866 consensus_queue.cc:237] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:16:57.728910 12334 consensus_queue.cc:237] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
W20240502 14:16:57.732141 26493 leader_election.cc:334] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:16:57.735847 26280 catalog_manager.cc:5484] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a reported cstate change: term changed from 1 to 2, leader changed from c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) to ba187e38a2754d60a12de9ac4856430a (127.19.94.194). New cstate: current_term: 2 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:57.737859 29438 raft_consensus.cc:2454] T f18990f33bf74c31a0faf5802cc1856d P 68a3c2b17ed94e06aba66ed4178132ab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4092902b3e674857a0147f1b4f2cdb86 in term 2.
I20240502 14:16:57.738418 29511 leader_election.cc:302] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, 68a3c2b17ed94e06aba66ed4178132ab; no voters: c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:16:57.742089 12024 raft_consensus.cc:2781] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:16:57.742290 12024 raft_consensus.cc:686] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 [term 2 LEADER]: Becoming Leader. State: Replica: 4092902b3e674857a0147f1b4f2cdb86, State: Running, Role: LEADER
I20240502 14:16:57.742576 12024 consensus_queue.cc:237] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:16:57.745798 26280 catalog_manager.cc:5484] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a reported cstate change: term changed from 1 to 2, leader changed from c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) to ba187e38a2754d60a12de9ac4856430a (127.19.94.194). New cstate: current_term: 2 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:16:57.746434 26280 catalog_manager.cc:5484] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a reported cstate change: term changed from 1 to 2, leader changed from c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) to ba187e38a2754d60a12de9ac4856430a (127.19.94.194). New cstate: current_term: 2 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
W20240502 14:16:57.750542 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:57.756669 29489 maintenance_manager.cc:419] P 68a3c2b17ed94e06aba66ed4178132ab: Scheduling FlushMRSOp(533b42de2de14abc96c00a928d5deed1): perf score=0.033350
I20240502 14:16:57.780247 26280 catalog_manager.cc:5484] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: term changed from 1 to 2, leader changed from c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) to 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200). New cstate: current_term: 2 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:16:57.785785 29621 maintenance_manager.cc:419] P 4092902b3e674857a0147f1b4f2cdb86: Scheduling FlushMRSOp(c586354e10cb4bedbbb20bd103ad477e): perf score=0.033393
I20240502 14:16:57.803875 26280 catalog_manager.cc:5484] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: term changed from 1 to 2, leader changed from c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) to e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196). New cstate: current_term: 2 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
W20240502 14:16:57.817241 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:16:57.822104 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:57.863703 26686 raft_consensus.cc:1261] T 46aaa6186d254ff7bd493bd85479015d P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 2 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:16:57.864709 12304 consensus_queue.cc:1035] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:57.893456 26688 raft_consensus.cc:1261] T d6c8fa23fffe491fbfcec03afdb2052a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 2 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:16:57.894299 11696 consensus_queue.cc:1035] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:57.912454 29574 raft_consensus.cc:1261] T 33a3639c26bf4b35a6e15d6ef173751c P 4092902b3e674857a0147f1b4f2cdb86 [term 2 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:16:57.913208 12344 consensus_queue.cc:1035] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:16:57.930225 29111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.194:36025, user_credentials={real_user=slave}} blocked reactor thread for 74675.2us
I20240502 14:16:57.937626 29155 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:16:57.955149 29421 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:16:57.969085 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:57.995831 29378 consensus_peers.cc:485] T cf808a060536445495a70de638438f87 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:58.000804 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:16:58.006623 26538 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:16:58.021085 26798 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:16:58.021085 26406 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:16:58.027906 26424 raft_consensus.cc:1261] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:16:58.040313 12302 consensus_queue.cc:1035] T cf808a060536445495a70de638438f87 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:58.050307 26424 raft_consensus.cc:1261] T 2e59791e95cc452ea30d1ed8bb6178ae P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:16:58.051009 12299 consensus_queue.cc:1035] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:58.061792 26669 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240502 14:16:58.077929 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:58.079416 29512 maintenance_manager.cc:643] P 4092902b3e674857a0147f1b4f2cdb86: FlushMRSOp(c586354e10cb4bedbbb20bd103ad477e) complete. Timing: real 0.293s user 0.053s sys 0.018s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":13115,"dirs.run_cpu_time_us":240,"dirs.run_wall_time_us":6747,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1556,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":72570,"threads_started":2,"wal-append.queue_time_us":53628}
I20240502 14:16:58.086208 29621 maintenance_manager.cc:419] P 4092902b3e674857a0147f1b4f2cdb86: Scheduling FlushMRSOp(62218b2a597e4fb188ced6d2b9a83a59): perf score=0.033428
I20240502 14:16:58.145087 29438 raft_consensus.cc:1261] T f18990f33bf74c31a0faf5802cc1856d P 68a3c2b17ed94e06aba66ed4178132ab [term 2 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:16:58.149845 12346 consensus_queue.cc:1035] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20240502 14:16:58.157407 29443 raft_consensus.cc:1261] T abdd0f01d8d5453793c7e9461af10516 P 68a3c2b17ed94e06aba66ed4178132ab [term 2 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:16:58.158305 12015 consensus_queue.cc:1035] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:58.159880 26424 raft_consensus.cc:1261] T 924ce5625bf34a739169b005ee4bedc2 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240502 14:16:58.160578 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:58.162345 12307 consensus_queue.cc:1035] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:16:58.171308 26556 raft_consensus.cc:1261] T 13136deb9046440f88bf3471ff42312c P ba187e38a2754d60a12de9ac4856430a [term 2 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:16:58.171998 11696 consensus_queue.cc:1035] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:58.173429 26424 raft_consensus.cc:1261] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:16:58.175683 11866 consensus_queue.cc:1035] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:58.187804 26425 raft_consensus.cc:1261] T 56285a2fdcdd47e9b9c81b4b8993c4aa P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:16:58.188679 12344 consensus_queue.cc:1035] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:16:58.192571 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:58.195854 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:58.197427 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:58.202792 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:58.265305 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:16:58.271144 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:16:58.280241 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:16:58.283067 26556 raft_consensus.cc:1261] T abdd0f01d8d5453793c7e9461af10516 P ba187e38a2754d60a12de9ac4856430a [term 2 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:16:58.283864 12015 consensus_queue.cc:1035] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:16:58.288522 29380 maintenance_manager.cc:643] P 68a3c2b17ed94e06aba66ed4178132ab: FlushMRSOp(533b42de2de14abc96c00a928d5deed1) complete. Timing: real 0.531s user 0.087s sys 0.015s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":11538,"dirs.run_cpu_time_us":246,"dirs.run_wall_time_us":1443,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1276,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":34482,"threads_started":2,"wal-append.queue_time_us":18504}
I20240502 14:16:58.294267 29489 maintenance_manager.cc:419] P 68a3c2b17ed94e06aba66ed4178132ab: Scheduling UndoDeltaBlockGCOp(533b42de2de14abc96c00a928d5deed1): 4838 bytes on disk
I20240502 14:16:58.295235 29380 maintenance_manager.cc:643] P 68a3c2b17ed94e06aba66ed4178132ab: UndoDeltaBlockGCOp(533b42de2de14abc96c00a928d5deed1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
I20240502 14:16:58.321444 26424 raft_consensus.cc:1261] T 3f6d55d89e55425183da73dbb3f4175f P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:16:58.322396 11866 consensus_queue.cc:1035] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:16:58.394529 26686 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1fb1ef75829641a89c4cac539f431d80"
dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48258
I20240502 14:16:58.394766 26686 raft_consensus.cc:595] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:16:58.412323 29512 maintenance_manager.cc:643] P 4092902b3e674857a0147f1b4f2cdb86: FlushMRSOp(62218b2a597e4fb188ced6d2b9a83a59) complete. Timing: real 0.324s user 0.060s sys 0.019s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":162,"dirs.run_wall_time_us":1398,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":20488,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":21,"peak_mem_usage":0,"rows_written":635,"thread_start_us":11395,"threads_started":1,"wal-append.queue_time_us":11499}
I20240502 14:16:58.415273 29621 maintenance_manager.cc:419] P 4092902b3e674857a0147f1b4f2cdb86: Scheduling UndoDeltaBlockGCOp(c586354e10cb4bedbbb20bd103ad477e): 4840 bytes on disk
I20240502 14:16:58.417327 29512 maintenance_manager.cc:643] P 4092902b3e674857a0147f1b4f2cdb86: UndoDeltaBlockGCOp(c586354e10cb4bedbbb20bd103ad477e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4}
I20240502 14:16:58.421641 29621 maintenance_manager.cc:419] P 4092902b3e674857a0147f1b4f2cdb86: Scheduling UndoDeltaBlockGCOp(62218b2a597e4fb188ced6d2b9a83a59): 4582 bytes on disk
I20240502 14:16:58.423081 29512 maintenance_manager.cc:643] P 4092902b3e674857a0147f1b4f2cdb86: UndoDeltaBlockGCOp(62218b2a597e4fb188ced6d2b9a83a59) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
I20240502 14:16:58.494889 12307 raft_consensus.cc:979] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: : Instructing follower 89af7f423cf44717a08d6b142f0ed49d to start an election
I20240502 14:16:58.495069 12307 raft_consensus.cc:1067] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Signalling peer 89af7f423cf44717a08d6b142f0ed49d to start an election
I20240502 14:16:58.498528 26427 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1fb1ef75829641a89c4cac539f431d80"
dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
from {username='slave'} at 127.19.94.195:34637
I20240502 14:16:58.498778 26427 raft_consensus.cc:482] T 1fb1ef75829641a89c4cac539f431d80 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:16:58.498925 26427 raft_consensus.cc:3037] T 1fb1ef75829641a89c4cac539f431d80 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:58.502838 26427 raft_consensus.cc:504] T 1fb1ef75829641a89c4cac539f431d80 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:16:58.504276 26686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fb1ef75829641a89c4cac539f431d80" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:16:58.504526 26686 raft_consensus.cc:3032] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:16:58.504606 26686 raft_consensus.cc:729] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Running, Role: LEADER
I20240502 14:16:58.504731 29175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fb1ef75829641a89c4cac539f431d80" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:16:58.504933 29175 raft_consensus.cc:3037] T 1fb1ef75829641a89c4cac539f431d80 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:58.505437 26686 consensus_queue.cc:260] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:16:58.506175 26686 raft_consensus.cc:3037] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:58.507874 29175 raft_consensus.cc:2454] T 1fb1ef75829641a89c4cac539f431d80 P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 2.
I20240502 14:16:58.509192 26686 raft_consensus.cc:2454] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 2.
I20240502 14:16:58.511036 26362 leader_election.cc:302] T 1fb1ef75829641a89c4cac539f431d80 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 69ab78b2a1b74d2bbf2cdeafa0a4af23, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:16:58.511384 12388 raft_consensus.cc:2781] T 1fb1ef75829641a89c4cac539f431d80 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:16:58.511543 12388 raft_consensus.cc:686] T 1fb1ef75829641a89c4cac539f431d80 P 89af7f423cf44717a08d6b142f0ed49d [term 2 LEADER]: Becoming Leader. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Running, Role: LEADER
I20240502 14:16:58.511843 12388 consensus_queue.cc:237] T 1fb1ef75829641a89c4cac539f431d80 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:16:58.522576 29574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fb1ef75829641a89c4cac539f431d80" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "4092902b3e674857a0147f1b4f2cdb86"
I20240502 14:16:58.522823 29574 raft_consensus.cc:3037] T 1fb1ef75829641a89c4cac539f431d80 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:16:58.529155 26427 leader_election.cc:288] T 1fb1ef75829641a89c4cac539f431d80 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 election: Requested vote from peers 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097)
I20240502 14:16:58.539464 29574 raft_consensus.cc:2454] T 1fb1ef75829641a89c4cac539f431d80 P 4092902b3e674857a0147f1b4f2cdb86 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 2.
I20240502 14:16:58.546388 26280 catalog_manager.cc:5484] T 1fb1ef75829641a89c4cac539f431d80 P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: term changed from 1 to 2, leader changed from 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) to 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193). New cstate: current_term: 2 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
W20240502 14:16:58.649844 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:16:58.700969 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:16:58.766136 29155 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:16:58.770403 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:16:58.773685 29421 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:58.816704 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:16:58.855391 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:16:58.860909 26538 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:16:58.861397 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:16:58.874431 26798 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:16:58.901008 26688 raft_consensus.cc:1261] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 2 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:16:58.901650 12108 consensus_queue.cc:1035] T 1fb1ef75829641a89c4cac539f431d80 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 14:16:58.905992 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:16:58.916077 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:16:58.938997 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:16:58.939674 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:16:58.967698 29378 consensus_peers.cc:485] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:16:58.986152 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:16:59.001498 29574 raft_consensus.cc:1261] T 1fb1ef75829641a89c4cac539f431d80 P 4092902b3e674857a0147f1b4f2cdb86 [term 2 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:16:59.002043 12108 consensus_queue.cc:1035] T 1fb1ef75829641a89c4cac539f431d80 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 14:16:59.003127 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:16:59.009681 29175 raft_consensus.cc:1261] T 1fb1ef75829641a89c4cac539f431d80 P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:16:59.010391 12108 consensus_queue.cc:1035] T 1fb1ef75829641a89c4cac539f431d80 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 14:16:59.038188 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:16:59.048097 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:16:59.058884 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:16:59.068513 29378 consensus_peers.cc:485] T c32ea761f2834d758529890e5b496f88 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:16:59.075416 29378 consensus_peers.cc:485] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:16:59.084076 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:16:59.111824 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:16:59.113801 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:16:59.128623 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:16:59.146016 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:16:59.178320 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:16:59.229184 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:16:59.238322 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:16:59.244271 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:16:59.252588 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:16:59.278925 29378 consensus_peers.cc:485] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:16:59.286784 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:16:59.295922 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:16:59.343271 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:16:59.406833 29155 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240502 14:16:59.417523 29378 consensus_peers.cc:485] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:16:59.417686 29378 consensus_peers.cc:485] T a511c4f2b4c14f4a90872bf0093c035c P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:16:59.486788 29378 consensus_peers.cc:485] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:16:59.501852 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:16:59.566251 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:16:59.583649 29421 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:59.587944 26798 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:16:59.592943 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:16:59.619801 26538 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240502 14:16:59.655905 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:00.058887 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:00.080593 26538 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:17:00.103339 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:17:00.111865 29155 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:17:00.121605 29421 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:17:00.136099 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:00.152282 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:17:00.207952 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:17:00.216719 26798 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:17:00.372099 26558 consensus_queue.cc:237] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
W20240502 14:17:00.406666 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:17:00.407815 26426 raft_consensus.cc:1261] T 04f3eff43b544f33b020bd74e4331dea P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:00.407821 26686 raft_consensus.cc:1261] T 04f3eff43b544f33b020bd74e4331dea P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:00.408432 12330 consensus_queue.cc:1035] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:00.408885 12344 consensus_queue.cc:1035] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:00.412421 12344 raft_consensus.cc:2932] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:17:00.414513 26426 raft_consensus.cc:2932] T 04f3eff43b544f33b020bd74e4331dea P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:17:00.420312 26284 catalog_manager.cc:5484] T 04f3eff43b544f33b020bd74e4331dea P ba187e38a2754d60a12de9ac4856430a reported cstate change: config changed from index 4 to 5, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) evicted. New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:17:00.421782 26688 raft_consensus.cc:2932] T 04f3eff43b544f33b020bd74e4331dea P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
W20240502 14:17:00.432619 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:17:00.448506 29554 tablet_service.cc:1514] Processing DeleteTablet for tablet 04f3eff43b544f33b020bd74e4331dea with delete_type TABLET_DATA_TOMBSTONED (TS 4092902b3e674857a0147f1b4f2cdb86 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:36398
I20240502 14:17:00.449177 12641 tablet_replica.cc:330] T 04f3eff43b544f33b020bd74e4331dea P 4092902b3e674857a0147f1b4f2cdb86: stopping tablet replica
I20240502 14:17:00.449918 12641 raft_consensus.cc:2229] T 04f3eff43b544f33b020bd74e4331dea P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:17:00.450290 12641 raft_consensus.cc:2258] T 04f3eff43b544f33b020bd74e4331dea P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:17:00.451117 12641 ts_tablet_manager.cc:1875] T 04f3eff43b544f33b020bd74e4331dea P 4092902b3e674857a0147f1b4f2cdb86: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:17:00.457877 12641 ts_tablet_manager.cc:1888] T 04f3eff43b544f33b020bd74e4331dea P 4092902b3e674857a0147f1b4f2cdb86: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:17:00.458161 12641 log.cc:1196] T 04f3eff43b544f33b020bd74e4331dea P 4092902b3e674857a0147f1b4f2cdb86: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-7/wal/wals/04f3eff43b544f33b020bd74e4331dea
I20240502 14:17:00.460780 26272 catalog_manager.cc:4830] TS 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097): tablet 04f3eff43b544f33b020bd74e4331dea (table rebalancer_extra_table_50 [id=183571cb1d8d40aaa35266ced8ac1c76]) successfully deleted
W20240502 14:17:00.488780 29378 consensus_peers.cc:485] T cf808a060536445495a70de638438f87 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:17:00.519541 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:17:00.538292 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:17:00.552139 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:17:00.553503 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:00.590696 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:17:00.633911 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:17:00.653326 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:17:00.696251 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:17:00.750437 29155 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:17:00.771395 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:00.867722 29421 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:17:00.869396 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:17:00.880226 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:17:00.889194 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:17:00.904752 26538 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:17:00.967698 26798 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240502 14:17:01.093886 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:01.132505 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:01.214726 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:01.267199 29378 consensus_peers.cc:485] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:01.293978 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:01.305433 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:01.320619 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:01.364565 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:01.372340 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:01.455241 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:01.491353 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:01.496683 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:01.514410 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:01.525820 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:17:01.529695 29421 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:17:01.538501 29511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.196:32843, user_credentials={real_user=slave}} blocked reactor thread for 70375.9us
I20240502 14:17:01.553100 29155 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240502 14:17:01.569346 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:01.576692 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:17:01.599318 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:01.614353 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:01.619921 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:17:01.642686 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:17:01.649037 29378 consensus_peers.cc:485] T c32ea761f2834d758529890e5b496f88 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:17:01.651993 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:17:01.690438 29378 consensus_peers.cc:485] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:01.694973 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:01.704756 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:01.739042 29378 consensus_peers.cc:485] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:01.747705 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:17:01.752188 26538 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240502 14:17:01.764848 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:01.778343 29510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:17:01.785715 26754 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240502 14:17:01.802487 26493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240502 14:17:01.804018 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:01.823217 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:17:01.825749 26798 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240502 14:17:01.850488 26624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 44 similar messages]
W20240502 14:17:01.853579 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:01.886181 26362 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240502 14:17:01.908143 29111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 55 similar messages]
W20240502 14:17:01.941123 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:17:01.989295 26819 consensus_queue.cc:237] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:17:01.991238 26558 raft_consensus.cc:1261] T a42e68ab03f84d4b92893006fcf6f229 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:01.991240 29440 raft_consensus.cc:1261] T a42e68ab03f84d4b92893006fcf6f229 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:01.991775 12768 consensus_queue.cc:1035] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:01.992125 12015 consensus_queue.cc:1035] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:01.995013 12768 raft_consensus.cc:2932] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:17:01.995783 26558 raft_consensus.cc:2932] T a42e68ab03f84d4b92893006fcf6f229 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:17:01.998769 29440 raft_consensus.cc:2932] T a42e68ab03f84d4b92893006fcf6f229 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:17:02.003294 26280 catalog_manager.cc:5484] T a42e68ab03f84d4b92893006fcf6f229 P ba187e38a2754d60a12de9ac4856430a reported cstate change: config changed from index 4 to 5, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) evicted. New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:17:02.015071 26406 tablet_service.cc:1514] Processing DeleteTablet for tablet a42e68ab03f84d4b92893006fcf6f229 with delete_type TABLET_DATA_TOMBSTONED (TS 89af7f423cf44717a08d6b142f0ed49d not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:37422
I20240502 14:17:02.016124 12797 tablet_replica.cc:330] T a42e68ab03f84d4b92893006fcf6f229 P 89af7f423cf44717a08d6b142f0ed49d: stopping tablet replica
I20240502 14:17:02.016479 12797 raft_consensus.cc:2229] T a42e68ab03f84d4b92893006fcf6f229 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:17:02.016856 12797 raft_consensus.cc:2258] T a42e68ab03f84d4b92893006fcf6f229 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:17:02.017568 12797 ts_tablet_manager.cc:1875] T a42e68ab03f84d4b92893006fcf6f229 P 89af7f423cf44717a08d6b142f0ed49d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:17:02.025786 12797 ts_tablet_manager.cc:1888] T a42e68ab03f84d4b92893006fcf6f229 P 89af7f423cf44717a08d6b142f0ed49d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:17:02.026059 12797 log.cc:1196] T a42e68ab03f84d4b92893006fcf6f229 P 89af7f423cf44717a08d6b142f0ed49d: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/a42e68ab03f84d4b92893006fcf6f229
I20240502 14:17:02.026945 26271 catalog_manager.cc:4830] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): tablet a42e68ab03f84d4b92893006fcf6f229 (table rebalancer_extra_table_54 [id=36399b8bbc064edb98ef3de360e0e9bf]) successfully deleted
W20240502 14:17:02.034185 29378 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 86 similar messages]
W20240502 14:17:02.035322 29378 consensus_peers.cc:485] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:02.075799 29378 consensus_peers.cc:485] T a511c4f2b4c14f4a90872bf0093c035c P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:02.090853 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:02.098268 29378 consensus_peers.cc:485] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:17:02.441032 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:02.449265 29155 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:17:02.502151 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:17:02.555244 29421 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:17:02.579118 26538 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:17:02.584198 26798 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240502 14:17:02.656788 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:02.675882 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:17:02.695197 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:17:02.815675 29574 consensus_queue.cc:237] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:17:02.817893 29440 raft_consensus.cc:1261] T 82e51aa5371e4731bc00daafc65c7a88 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:17:02.817911 26558 raft_consensus.cc:1261] T 82e51aa5371e4731bc00daafc65c7a88 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Refusing update from remote peer 4092902b3e674857a0147f1b4f2cdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:17:02.818439 12858 consensus_queue.cc:1035] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:17:02.818732 12758 consensus_queue.cc:1035] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:17:02.823755 12858 raft_consensus.cc:2932] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:17:02.824874 29440 raft_consensus.cc:2932] T 82e51aa5371e4731bc00daafc65c7a88 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:17:02.833397 26281 catalog_manager.cc:5484] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 reported cstate change: config changed from index 2 to 3, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) evicted. New cstate: current_term: 1 leader_uuid: "4092902b3e674857a0147f1b4f2cdb86" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:17:02.838272 26558 raft_consensus.cc:2932] T 82e51aa5371e4731bc00daafc65c7a88 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:17:02.846204 29155 tablet_service.cc:1514] Processing DeleteTablet for tablet 82e51aa5371e4731bc00daafc65c7a88 with delete_type TABLET_DATA_TOMBSTONED (TS 1cb67158133e4648a6eceba32d4dbba3 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:55202
I20240502 14:17:02.846909 12888 tablet_replica.cc:330] T 82e51aa5371e4731bc00daafc65c7a88 P 1cb67158133e4648a6eceba32d4dbba3: stopping tablet replica
I20240502 14:17:02.847250 12888 raft_consensus.cc:2229] T 82e51aa5371e4731bc00daafc65c7a88 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:17:02.847573 12888 raft_consensus.cc:2258] T 82e51aa5371e4731bc00daafc65c7a88 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:17:02.848335 12888 ts_tablet_manager.cc:1875] T 82e51aa5371e4731bc00daafc65c7a88 P 1cb67158133e4648a6eceba32d4dbba3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 14:17:02.856757 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:17:02.857412 12888 ts_tablet_manager.cc:1888] T 82e51aa5371e4731bc00daafc65c7a88 P 1cb67158133e4648a6eceba32d4dbba3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:17:02.857561 12888 log.cc:1196] T 82e51aa5371e4731bc00daafc65c7a88 P 1cb67158133e4648a6eceba32d4dbba3: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal/wals/82e51aa5371e4731bc00daafc65c7a88
I20240502 14:17:02.858330 26271 catalog_manager.cc:4830] TS 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): tablet 82e51aa5371e4731bc00daafc65c7a88 (table rebalancer_extra_table_14 [id=312a6520544b4149a5813f3f7652f51f]) successfully deleted
W20240502 14:17:02.873409 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:02.937594 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:03.063690 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:03.093274 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:17:03.121671 29155 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240502 14:17:03.124451 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:03.143203 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:03.173800 29378 consensus_peers.cc:485] T cf808a060536445495a70de638438f87 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:03.197767 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:17:03.205883 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:03.278522 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:17:03.279527 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:17:03.305449 29421 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:17:03.359251 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:17:03.361160 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:17:03.368116 26538 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240502 14:17:03.398366 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:17:03.417259 26798 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240502 14:17:03.538956 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:03.625499 29378 consensus_peers.cc:485] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:03.732336 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:03.740934 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:03.782797 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:03.791852 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:03.802453 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:03.826757 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:03.841948 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:17:03.909698 29155 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240502 14:17:03.913331 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:03.917440 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:17:03.933760 29421 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:17:03.941869 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:03.947494 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:17:03.949695 26538 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240502 14:17:03.977706 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:17:03.984704 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:17:03.989282 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:17:04.004276 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:04.043865 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:04.045969 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:04.065965 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:04.082182 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:17:04.097144 26798 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240502 14:17:04.194458 29378 consensus_peers.cc:485] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:04.195608 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:04.198244 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:04.203217 29378 consensus_peers.cc:485] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:04.220259 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:04.278354 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:04.288745 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:04.370188 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:04.422166 29378 consensus_peers.cc:485] T c32ea761f2834d758529890e5b496f88 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:04.430253 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:04.543501 29378 consensus_peers.cc:485] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:04.555824 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:04.570145 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:17:04.723750 29155 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240502 14:17:04.777699 29378 consensus_peers.cc:485] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:04.779349 29378 consensus_peers.cc:485] T a511c4f2b4c14f4a90872bf0093c035c P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:17:04.782791 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:17:04.822302 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:04.835179 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:04.848986 29421 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:17:04.856707 26538 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:17:04.920451 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:17:04.921720 26798 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:17:05.009375 26818 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ce03ee354c54457487ea939301ccd90e"
dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44430
I20240502 14:17:05.009933 26818 raft_consensus.cc:595] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Received request to transfer leadership
W20240502 14:17:05.134342 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:05.247931 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:17:05.333971 12767 raft_consensus.cc:979] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27: : Instructing follower 68a3c2b17ed94e06aba66ed4178132ab to start an election
I20240502 14:17:05.334136 12767 raft_consensus.cc:1067] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Signalling peer 68a3c2b17ed94e06aba66ed4178132ab to start an election
I20240502 14:17:05.351374 29440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ce03ee354c54457487ea939301ccd90e"
dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
from {username='slave'} at 127.19.94.196:41215
I20240502 14:17:05.351615 29440 raft_consensus.cc:482] T ce03ee354c54457487ea939301ccd90e P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:17:05.351745 29440 raft_consensus.cc:3037] T ce03ee354c54457487ea939301ccd90e P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:17:05.355535 29440 raft_consensus.cc:504] T ce03ee354c54457487ea939301ccd90e P 68a3c2b17ed94e06aba66ed4178132ab [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:17:05.357419 29174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce03ee354c54457487ea939301ccd90e" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:17:05.357666 29174 raft_consensus.cc:3037] T ce03ee354c54457487ea939301ccd90e P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:17:05.357724 26685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce03ee354c54457487ea939301ccd90e" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:17:05.357929 26685 raft_consensus.cc:3037] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:17:05.360635 29174 raft_consensus.cc:2454] T ce03ee354c54457487ea939301ccd90e P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 2.
I20240502 14:17:05.360718 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce03ee354c54457487ea939301ccd90e" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
I20240502 14:17:05.360980 26818 raft_consensus.cc:3032] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:17:05.360958 26685 raft_consensus.cc:2454] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 2.
I20240502 14:17:05.361059 26818 raft_consensus.cc:729] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Running, Role: LEADER
I20240502 14:17:05.361362 29440 leader_election.cc:288] T ce03ee354c54457487ea939301ccd90e P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 2 election: Requested vote from peers e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:17:05.361500 29379 leader_election.cc:302] T ce03ee354c54457487ea939301ccd90e P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 68a3c2b17ed94e06aba66ed4178132ab, 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters:
I20240502 14:17:05.361408 26818 consensus_queue.cc:260] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:17:05.361830 12983 raft_consensus.cc:2781] T ce03ee354c54457487ea939301ccd90e P 68a3c2b17ed94e06aba66ed4178132ab [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:17:05.361951 12983 raft_consensus.cc:686] T ce03ee354c54457487ea939301ccd90e P 68a3c2b17ed94e06aba66ed4178132ab [term 2 LEADER]: Becoming Leader. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Running, Role: LEADER
I20240502 14:17:05.361970 26818 raft_consensus.cc:3037] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:17:05.362222 12983 consensus_queue.cc:237] T ce03ee354c54457487ea939301ccd90e P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:17:05.367614 26818 raft_consensus.cc:2454] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 2.
I20240502 14:17:05.370987 26282 catalog_manager.cc:5484] T ce03ee354c54457487ea939301ccd90e P 68a3c2b17ed94e06aba66ed4178132ab reported cstate change: term changed from 1 to 2, leader changed from e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) to 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199). New cstate: current_term: 2 leader_uuid: "68a3c2b17ed94e06aba66ed4178132ab" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:17:05.422572 29155 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240502 14:17:05.434592 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:05.445338 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:05.468355 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:05.491732 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:17:05.539781 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:05.568157 29421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240502 14:17:05.571879 29378 consensus_peers.cc:485] T cf808a060536445495a70de638438f87 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:17:05.581730 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:17:05.608690 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:17:05.612370 26538 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240502 14:17:05.619930 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:05.640766 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:17:05.644264 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 14:17:05.762799 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:17:05.793524 26818 raft_consensus.cc:1261] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:17:05.794082 12983 consensus_queue.cc:1035] T ce03ee354c54457487ea939301ccd90e P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 14:17:05.846495 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:05.890802 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:17:05.901352 26685 raft_consensus.cc:1261] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 2 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:17:05.902472 12983 consensus_queue.cc:1035] T ce03ee354c54457487ea939301ccd90e P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:17:05.943719 29176 raft_consensus.cc:1261] T ce03ee354c54457487ea939301ccd90e P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:17:05.944530 12983 consensus_queue.cc:1035] T ce03ee354c54457487ea939301ccd90e P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 14:17:06.023638 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:06.071202 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:06.159595 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:06.160476 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:17:06.175114 29155 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:17:06.226327 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:06.242090 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:17:06.260798 29421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240502 14:17:06.313356 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:17:06.327607 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:17:06.328011 26538 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:17:06.350395 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:17:06.366513 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:06.392896 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:06.398382 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:06.467844 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:06.468314 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:06.469368 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:06.497123 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:06.503826 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:06.524956 29378 consensus_peers.cc:485] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:06.552618 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:06.567188 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:06.570220 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:06.601802 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:06.648023 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:06.662648 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:06.688858 29378 consensus_peers.cc:485] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:06.697333 29378 consensus_peers.cc:485] T c32ea761f2834d758529890e5b496f88 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:06.712532 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:06.758968 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:06.823366 26493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 126 similar messages]
W20240502 14:17:06.844444 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:06.846975 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:06.852252 29378 consensus_peers.cc:485] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:17:06.860319 29155 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240502 14:17:06.879858 26754 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240502 14:17:06.884968 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:06.908046 29510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240502 14:17:06.916103 29421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:17:06.918080 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:17:06.928190 26362 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 14:17:06.934322 26624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 14:17:06.950318 26538 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:17:06.957185 26798 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:17:06.977813 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:17:06.982512 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:06.990180 29111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 14:17:07.056424 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:07.099195 29378 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 79 similar messages]
I20240502 14:17:07.101521 26685 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "25b56a948668442da540ca7a198f9fbb"
dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49988
I20240502 14:17:07.101751 26685 raft_consensus.cc:595] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Received request to transfer leadership
W20240502 14:17:07.215109 29378 consensus_peers.cc:485] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:07.221485 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:07.259313 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:07.294348 12987 raft_consensus.cc:979] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23: : Instructing follower e4cfb8fc5a474177af6848a6efa13a27 to start an election
I20240502 14:17:07.294549 12987 raft_consensus.cc:1067] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Signalling peer e4cfb8fc5a474177af6848a6efa13a27 to start an election
I20240502 14:17:07.295353 26815 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "25b56a948668442da540ca7a198f9fbb"
dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27"
from {username='slave'} at 127.19.94.195:42141
I20240502 14:17:07.295560 26815 raft_consensus.cc:482] T 25b56a948668442da540ca7a198f9fbb P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:17:07.295661 26815 raft_consensus.cc:3037] T 25b56a948668442da540ca7a198f9fbb P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:17:07.298600 26815 raft_consensus.cc:504] T 25b56a948668442da540ca7a198f9fbb P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:17:07.300724 29174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25b56a948668442da540ca7a198f9fbb" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:17:07.300997 29174 raft_consensus.cc:3037] T 25b56a948668442da540ca7a198f9fbb P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:17:07.316300 29174 raft_consensus.cc:2454] T 25b56a948668442da540ca7a198f9fbb P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 2.
I20240502 14:17:07.317174 26815 leader_election.cc:288] T 25b56a948668442da540ca7a198f9fbb P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 election: Requested vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:17:07.317785 26689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25b56a948668442da540ca7a198f9fbb" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23"
I20240502 14:17:07.318037 26689 raft_consensus.cc:3032] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:17:07.318135 26689 raft_consensus.cc:729] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Running, Role: LEADER
I20240502 14:17:07.318459 26689 consensus_queue.cc:260] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:17:07.318835 26689 raft_consensus.cc:3037] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:17:07.321472 26689 raft_consensus.cc:2454] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 2.
I20240502 14:17:07.321946 26427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25b56a948668442da540ca7a198f9fbb" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:17:07.322198 26427 raft_consensus.cc:3037] T 25b56a948668442da540ca7a198f9fbb P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:17:07.322306 26755 leader_election.cc:302] T 25b56a948668442da540ca7a198f9fbb P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 69ab78b2a1b74d2bbf2cdeafa0a4af23, e4cfb8fc5a474177af6848a6efa13a27; no voters:
I20240502 14:17:07.322767 13182 raft_consensus.cc:2781] T 25b56a948668442da540ca7a198f9fbb P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:17:07.322891 13182 raft_consensus.cc:686] T 25b56a948668442da540ca7a198f9fbb P e4cfb8fc5a474177af6848a6efa13a27 [term 2 LEADER]: Becoming Leader. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Running, Role: LEADER
I20240502 14:17:07.323148 13182 consensus_queue.cc:237] T 25b56a948668442da540ca7a198f9fbb P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:17:07.325254 26427 raft_consensus.cc:2454] T 25b56a948668442da540ca7a198f9fbb P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 2.
I20240502 14:17:07.346802 26276 catalog_manager.cc:5484] T 25b56a948668442da540ca7a198f9fbb P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: term changed from 1 to 2, leader changed from 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) to e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196). New cstate: current_term: 2 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
W20240502 14:17:07.484539 29378 consensus_peers.cc:485] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:07.492992 29378 consensus_peers.cc:485] T a511c4f2b4c14f4a90872bf0093c035c P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:07.573684 29378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.194:36025, user_credentials={real_user=slave}} blocked reactor thread for 70415.8us
W20240502 14:17:07.626756 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:07.614156 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:07.695363 29510 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.197:38715, user_credentials={real_user=slave}} blocked reactor thread for 130119us
I20240502 14:17:07.726694 26689 raft_consensus.cc:1261] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 2 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:17:07.727476 13216 consensus_queue.cc:1035] T 25b56a948668442da540ca7a198f9fbb P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 14:17:07.765514 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:07.769102 26425 raft_consensus.cc:1261] T 25b56a948668442da540ca7a198f9fbb P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:17:07.769855 13216 consensus_queue.cc:1035] T 25b56a948668442da540ca7a198f9fbb P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:07.807747 29174 raft_consensus.cc:1261] T 25b56a948668442da540ca7a198f9fbb P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20240502 14:17:07.808441 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:17:07.809653 13216 consensus_queue.cc:1035] T 25b56a948668442da540ca7a198f9fbb P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:07.875156 29155 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240502 14:17:07.894850 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:17:07.904563 29421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:17:07.923766 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:07.924083 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:17:07.962162 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240502 14:17:07.995844 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:17:08.026654 26538 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:17:08.062608 26798 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240502 14:17:08.093904 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:08.234535 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:08.252974 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:17:08.258210 26815 consensus_queue.cc:237] T 25b56a948668442da540ca7a198f9fbb P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:17:08.260537 26425 raft_consensus.cc:1261] T 25b56a948668442da540ca7a198f9fbb P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:17:08.260598 29174 raft_consensus.cc:1261] T 25b56a948668442da540ca7a198f9fbb P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:17:08.261160 13182 consensus_queue.cc:1035] T 25b56a948668442da540ca7a198f9fbb P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:08.261690 13216 consensus_queue.cc:1035] T 25b56a948668442da540ca7a198f9fbb P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:08.264995 13216 raft_consensus.cc:2932] T 25b56a948668442da540ca7a198f9fbb P e4cfb8fc5a474177af6848a6efa13a27 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:17:08.265636 26425 raft_consensus.cc:2932] T 25b56a948668442da540ca7a198f9fbb P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
W20240502 14:17:08.269384 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:17:08.273411 26277 catalog_manager.cc:5484] T 25b56a948668442da540ca7a198f9fbb P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: config changed from index 4 to 6, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New cstate: current_term: 2 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
I20240502 14:17:08.277500 29174 raft_consensus.cc:2932] T 25b56a948668442da540ca7a198f9fbb P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
W20240502 14:17:08.287111 29378 consensus_peers.cc:485] T cf808a060536445495a70de638438f87 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:17:08.288508 26669 tablet_service.cc:1514] Processing DeleteTablet for tablet 25b56a948668442da540ca7a198f9fbb with delete_type TABLET_DATA_TOMBSTONED (TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34150
I20240502 14:17:08.301203 13413 tablet_replica.cc:330] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23: stopping tablet replica
I20240502 14:17:08.301641 13413 raft_consensus.cc:2229] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:17:08.302037 13413 raft_consensus.cc:2258] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:17:08.303241 13413 ts_tablet_manager.cc:1875] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:17:08.314254 13413 ts_tablet_manager.cc:1888] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 14:17:08.314422 13413 log.cc:1196] T 25b56a948668442da540ca7a198f9fbb P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/wals/25b56a948668442da540ca7a198f9fbb
I20240502 14:17:08.315304 26272 catalog_manager.cc:4830] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): tablet 25b56a948668442da540ca7a198f9fbb (table rebalancer_extra_table_32 [id=eb9f13fa5bdc4cc3b3c074bc196dfb8c]) successfully deleted
W20240502 14:17:08.342744 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:08.351493 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:08.427067 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:08.427919 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:08.609544 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:08.686544 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:17:08.723969 29155 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:17:08.731930 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:08.765000 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:17:08.806257 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:17:08.806257 29421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:17:08.846582 26798 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:17:08.846777 26538 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240502 14:17:08.859688 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:08.860003 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:08.885358 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:08.891819 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:08.902937 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:08.913425 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:08.986821 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:08.995631 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:09.003903 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:09.005545 29378 consensus_peers.cc:485] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:09.052650 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:09.071893 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:09.096482 29378 consensus_peers.cc:485] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:09.099184 29378 consensus_peers.cc:485] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:09.131795 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:09.148084 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:09.150527 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:09.207984 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:09.268370 29378 consensus_peers.cc:485] T c32ea761f2834d758529890e5b496f88 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:09.277014 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:09.348264 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:09.353560 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:09.401710 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:09.430990 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:09.461871 29155 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240502 14:17:09.504220 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:17:09.508016 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:09.565084 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:17:09.604493 29421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:17:09.635582 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:17:09.672370 26538 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240502 14:17:09.704360 29378 consensus_peers.cc:485] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:09.713244 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:09.722751 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:17:09.724679 26798 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240502 14:17:09.766619 29378 consensus_peers.cc:485] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:09.790357 29378 consensus_peers.cc:485] T a511c4f2b4c14f4a90872bf0093c035c P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:09.857996 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:10.182297 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240502 14:17:10.193457 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:17:10.212625 29421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:17:10.217625 29155 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240502 14:17:10.228730 29510 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.198:43659, user_credentials={real_user=slave}} blocked reactor thread for 68988.1us
I20240502 14:17:10.262878 26798 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:17:10.265329 26538 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:17:10.299453 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:10.322566 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:17:10.329730 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:17:10.459864 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:10.461567 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:10.470696 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:10.484963 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:17:10.546893 26685 consensus_queue.cc:237] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:17:10.549441 29437 raft_consensus.cc:1261] T 451b303a32a3404d8d584203d6704b35 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:10.549466 26819 raft_consensus.cc:1261] T 451b303a32a3404d8d584203d6704b35 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:17:10.550242 13624 consensus_queue.cc:1035] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:17:10.550631 12987 consensus_queue.cc:1035] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 14:17:10.552866 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:17:10.554934 13627 raft_consensus.cc:2932] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:17:10.556195 26819 raft_consensus.cc:2932] T 451b303a32a3404d8d584203d6704b35 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:17:10.556066 29437 raft_consensus.cc:2932] T 451b303a32a3404d8d584203d6704b35 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:17:10.568238 26285 catalog_manager.cc:5484] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: config changed from index 4 to 5, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) evicted. New cstate: current_term: 1 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:17:10.581239 29155 tablet_service.cc:1514] Processing DeleteTablet for tablet 451b303a32a3404d8d584203d6704b35 with delete_type TABLET_DATA_TOMBSTONED (TS 1cb67158133e4648a6eceba32d4dbba3 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:55202
I20240502 14:17:10.581900 13637 tablet_replica.cc:330] T 451b303a32a3404d8d584203d6704b35 P 1cb67158133e4648a6eceba32d4dbba3: stopping tablet replica
I20240502 14:17:10.582158 13637 raft_consensus.cc:2229] T 451b303a32a3404d8d584203d6704b35 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:17:10.582362 13637 raft_consensus.cc:2258] T 451b303a32a3404d8d584203d6704b35 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:17:10.582916 13637 ts_tablet_manager.cc:1875] T 451b303a32a3404d8d584203d6704b35 P 1cb67158133e4648a6eceba32d4dbba3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:17:10.588449 13637 ts_tablet_manager.cc:1888] T 451b303a32a3404d8d584203d6704b35 P 1cb67158133e4648a6eceba32d4dbba3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:17:10.588610 13637 log.cc:1196] T 451b303a32a3404d8d584203d6704b35 P 1cb67158133e4648a6eceba32d4dbba3: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-4/wal/wals/451b303a32a3404d8d584203d6704b35
I20240502 14:17:10.589401 26271 catalog_manager.cc:4830] TS 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): tablet 451b303a32a3404d8d584203d6704b35 (table rebalancer_extra_table_34 [id=b4142465f8a74e2baf1a95d85f3fd781]) successfully deleted
W20240502 14:17:10.639175 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:10.812227 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:10.862387 29378 consensus_peers.cc:485] T cf808a060536445495a70de638438f87 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:10.908282 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:17:10.908504 29155 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:17:10.919626 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:17:10.935137 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:10.950328 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:17:10.953294 26538 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240502 14:17:10.962357 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:11.009696 26493 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.197:38715, user_credentials={real_user=slave}} blocked reactor thread for 64007.6us
I20240502 14:17:11.010078 29421 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240502 14:17:11.022202 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:11.023519 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:11.081009 26798 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240502 14:17:11.153759 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:11.235888 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:11.239538 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:17:11.249835 26427 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9f3cbedfa1da42698615a544d9dc3276"
dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43262
I20240502 14:17:11.250180 26427 raft_consensus.cc:595] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Received request to transfer leadership
W20240502 14:17:11.258615 29378 consensus_peers.cc:485] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:11.422639 29378 consensus_peers.cc:485] T c32ea761f2834d758529890e5b496f88 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:11.427045 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:11.429132 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:11.480830 29378 consensus_peers.cc:485] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:11.487540 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:11.519924 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:11.530202 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:11.545190 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:17:11.557930 13301 raft_consensus.cc:979] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d: : Instructing follower 68a3c2b17ed94e06aba66ed4178132ab to start an election
I20240502 14:17:11.558120 13301 raft_consensus.cc:1067] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Signalling peer 68a3c2b17ed94e06aba66ed4178132ab to start an election
I20240502 14:17:11.572563 29440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9f3cbedfa1da42698615a544d9dc3276"
dest_uuid: "68a3c2b17ed94e06aba66ed4178132ab"
from {username='slave'} at 127.19.94.193:53537
I20240502 14:17:11.572865 29440 raft_consensus.cc:482] T 9f3cbedfa1da42698615a544d9dc3276 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:17:11.573002 29440 raft_consensus.cc:3037] T 9f3cbedfa1da42698615a544d9dc3276 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:17:11.589114 29440 raft_consensus.cc:504] T 9f3cbedfa1da42698615a544d9dc3276 P 68a3c2b17ed94e06aba66ed4178132ab [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
W20240502 14:17:11.590005 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:17:11.590451 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f3cbedfa1da42698615a544d9dc3276" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:17:11.590693 26558 raft_consensus.cc:3037] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:17:11.593691 26558 raft_consensus.cc:2454] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 2.
I20240502 14:17:11.600605 29440 leader_election.cc:288] T 9f3cbedfa1da42698615a544d9dc3276 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 2 election: Requested vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:17:11.601203 29175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f3cbedfa1da42698615a544d9dc3276" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:17:11.601507 29175 raft_consensus.cc:3037] T 9f3cbedfa1da42698615a544d9dc3276 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:17:11.604733 26427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f3cbedfa1da42698615a544d9dc3276" candidate_uuid: "68a3c2b17ed94e06aba66ed4178132ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "89af7f423cf44717a08d6b142f0ed49d"
I20240502 14:17:11.604988 26427 raft_consensus.cc:3032] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:17:11.605113 26427 raft_consensus.cc:729] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Running, Role: LEADER
I20240502 14:17:11.605401 26427 consensus_queue.cc:260] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:17:11.605788 26427 raft_consensus.cc:3037] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:17:11.606051 26558 raft_consensus.cc:1226] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a [term 2 FOLLOWER]: Rejecting Update request from peer 89af7f423cf44717a08d6b142f0ed49d for earlier term 1. Current term is 2. Ops: []
I20240502 14:17:11.608973 29175 raft_consensus.cc:2454] T 9f3cbedfa1da42698615a544d9dc3276 P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 2.
W20240502 14:17:11.609587 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:17:11.609616 29378 leader_election.cc:302] T 9f3cbedfa1da42698615a544d9dc3276 P 68a3c2b17ed94e06aba66ed4178132ab [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 68a3c2b17ed94e06aba66ed4178132ab, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:17:11.610033 13192 raft_consensus.cc:2781] T 9f3cbedfa1da42698615a544d9dc3276 P 68a3c2b17ed94e06aba66ed4178132ab [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:17:11.610227 13192 raft_consensus.cc:686] T 9f3cbedfa1da42698615a544d9dc3276 P 68a3c2b17ed94e06aba66ed4178132ab [term 2 LEADER]: Becoming Leader. State: Replica: 68a3c2b17ed94e06aba66ed4178132ab, State: Running, Role: LEADER
I20240502 14:17:11.610766 26427 raft_consensus.cc:2454] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a3c2b17ed94e06aba66ed4178132ab in term 2.
I20240502 14:17:11.610641 13192 consensus_queue.cc:237] T 9f3cbedfa1da42698615a544d9dc3276 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
W20240502 14:17:11.614302 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:17:11.632984 26278 catalog_manager.cc:5484] T 9f3cbedfa1da42698615a544d9dc3276 P 68a3c2b17ed94e06aba66ed4178132ab reported cstate change: term changed from 1 to 2, leader changed from 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) to 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199). New cstate: current_term: 2 leader_uuid: "68a3c2b17ed94e06aba66ed4178132ab" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
W20240502 14:17:11.633944 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:11.654420 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:11.667541 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:11.668030 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:11.716661 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:11.717836 29378 consensus_peers.cc:485] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:11.780743 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:17:11.811808 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:11.817813 29155 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:17:11.822549 26406 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:11.860786 26493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 130 similar messages]
I20240502 14:17:11.861053 26798 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:17:11.882442 26538 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:17:11.898406 29421 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:17:11.912457 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:11.935802 26754 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240502 14:17:11.946846 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240502 14:17:11.948531 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:11.952636 26362 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 14:17:11.990212 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:11.990744 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:17:12.002617 26427 raft_consensus.cc:1261] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:17:12.003274 13353 consensus_queue.cc:1035] T 9f3cbedfa1da42698615a544d9dc3276 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 14:17:12.031257 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:12.050724 29111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 54 similar messages]
W20240502 14:17:12.073941 26493 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.198:43659, user_credentials={real_user=slave}} blocked reactor thread for 83735.8us
W20240502 14:17:12.082744 26624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240502 14:17:12.084318 29175 raft_consensus.cc:1261] T 9f3cbedfa1da42698615a544d9dc3276 P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:17:12.085269 13353 consensus_queue.cc:1035] T 9f3cbedfa1da42698615a544d9dc3276 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 14:17:12.086102 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:12.105306 26556 raft_consensus.cc:1261] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a [term 2 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:17:12.107969 13192 consensus_queue.cc:1035] T 9f3cbedfa1da42698615a544d9dc3276 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 14:17:12.154134 29378 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 14:17:12.156560 29378 consensus_peers.cc:485] T a511c4f2b4c14f4a90872bf0093c035c P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:12.176069 29378 consensus_peers.cc:485] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:12.206272 29510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:17:12.294016 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:12.330057 29378 consensus_peers.cc:485] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:12.574034 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:12.574383 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:12.617667 26798 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:17:12.642215 26538 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:17:12.643779 26406 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:17:12.653978 29421 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:17:12.657071 29155 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:17:12.702486 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240502 14:17:12.745105 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:12.838271 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:12.854171 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:17:12.859601 26558 consensus_queue.cc:237] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:17:12.874111 29175 raft_consensus.cc:1261] T 4bf6e7410afa4c048f5842ec2a426d77 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:17:12.874331 29440 raft_consensus.cc:1261] T 4bf6e7410afa4c048f5842ec2a426d77 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:17:12.875145 12769 consensus_queue.cc:1035] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:17:12.875555 13695 consensus_queue.cc:1035] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:17:12.879523 13695 raft_consensus.cc:2932] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:17:12.882154 29440 raft_consensus.cc:2932] T 4bf6e7410afa4c048f5842ec2a426d77 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:17:12.884657 29175 raft_consensus.cc:2932] T 4bf6e7410afa4c048f5842ec2a426d77 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:17:12.893325 26281 catalog_manager.cc:5484] T 4bf6e7410afa4c048f5842ec2a426d77 P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: config changed from index 6 to 7, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New cstate: current_term: 1 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:17:12.907541 26798 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bf6e7410afa4c048f5842ec2a426d77 with delete_type TABLET_DATA_TOMBSTONED (TS e4cfb8fc5a474177af6848a6efa13a27 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:56010
I20240502 14:17:12.908632 13881 tablet_replica.cc:330] T 4bf6e7410afa4c048f5842ec2a426d77 P e4cfb8fc5a474177af6848a6efa13a27: stopping tablet replica
I20240502 14:17:12.909018 13881 raft_consensus.cc:2229] T 4bf6e7410afa4c048f5842ec2a426d77 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:17:12.909348 13881 raft_consensus.cc:2258] T 4bf6e7410afa4c048f5842ec2a426d77 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:17:12.910615 13881 ts_tablet_manager.cc:1875] T 4bf6e7410afa4c048f5842ec2a426d77 P e4cfb8fc5a474177af6848a6efa13a27: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:17:12.928580 13881 ts_tablet_manager.cc:1888] T 4bf6e7410afa4c048f5842ec2a426d77 P e4cfb8fc5a474177af6848a6efa13a27: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 14:17:12.928752 13881 log.cc:1196] T 4bf6e7410afa4c048f5842ec2a426d77 P e4cfb8fc5a474177af6848a6efa13a27: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/wals/4bf6e7410afa4c048f5842ec2a426d77
I20240502 14:17:12.930418 26272 catalog_manager.cc:4830] TS e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): tablet 4bf6e7410afa4c048f5842ec2a426d77 (table rebalance_test_table_5 [id=8069f2786bf84bf4a012c26e82757ad4]) successfully deleted
W20240502 14:17:12.982385 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:12.987313 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:13.023730 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:13.105894 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:13.283569 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:13.285009 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:13.344369 29378 consensus_peers.cc:485] T cf808a060536445495a70de638438f87 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:17:13.408282 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:13.409122 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:17:13.409857 29421 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:17:13.422995 29155 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:17:13.453675 26406 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:13.464471 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:13.467895 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:17:13.496757 26538 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:17:13.547762 26798 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240502 14:17:13.575198 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:13.586408 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:13.641814 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:13.755884 26425 consensus_queue.cc:237] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:17:13.758462 29575 raft_consensus.cc:1261] T 0fc2e6c74dc34a45ae1ec46608642880 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:17:13.758823 29174 raft_consensus.cc:1261] T 0fc2e6c74dc34a45ae1ec46608642880 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:17:13.759213 13922 consensus_queue.cc:1035] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:17:13.759490 13301 consensus_queue.cc:1035] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:17:13.763748 13922 raft_consensus.cc:2932] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
W20240502 14:17:13.765619 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:13.769196 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:17:13.769210 29574 raft_consensus.cc:2932] T 0fc2e6c74dc34a45ae1ec46608642880 P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:17:13.776021 26281 catalog_manager.cc:5484] T 0fc2e6c74dc34a45ae1ec46608642880 P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: config changed from index 6 to 7, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) evicted. New cstate: current_term: 1 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:17:13.778733 29174 raft_consensus.cc:2932] T 0fc2e6c74dc34a45ae1ec46608642880 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ba187e38a2754d60a12de9ac4856430a (127.19.94.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
W20240502 14:17:13.782359 29378 consensus_peers.cc:485] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:17:13.784191 26538 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fc2e6c74dc34a45ae1ec46608642880 with delete_type TABLET_DATA_TOMBSTONED (TS ba187e38a2754d60a12de9ac4856430a not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51696
I20240502 14:17:13.784919 13981 tablet_replica.cc:330] T 0fc2e6c74dc34a45ae1ec46608642880 P ba187e38a2754d60a12de9ac4856430a: stopping tablet replica
I20240502 14:17:13.785178 13981 raft_consensus.cc:2229] T 0fc2e6c74dc34a45ae1ec46608642880 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:17:13.785495 13981 raft_consensus.cc:2258] T 0fc2e6c74dc34a45ae1ec46608642880 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:17:13.786604 13981 ts_tablet_manager.cc:1875] T 0fc2e6c74dc34a45ae1ec46608642880 P ba187e38a2754d60a12de9ac4856430a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:17:13.794904 13981 ts_tablet_manager.cc:1888] T 0fc2e6c74dc34a45ae1ec46608642880 P ba187e38a2754d60a12de9ac4856430a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 14:17:13.795025 13981 log.cc:1196] T 0fc2e6c74dc34a45ae1ec46608642880 P ba187e38a2754d60a12de9ac4856430a: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-1/wal/wals/0fc2e6c74dc34a45ae1ec46608642880
I20240502 14:17:13.795722 26271 catalog_manager.cc:4830] TS ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025): tablet 0fc2e6c74dc34a45ae1ec46608642880 (table rebalance_test_table_1 [id=d7c1bbb69be74fd3a8e809dd8649698d]) successfully deleted
W20240502 14:17:13.819694 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:13.846509 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:13.915665 29378 consensus_peers.cc:485] T c32ea761f2834d758529890e5b496f88 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:13.959038 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:13.988209 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:14.053297 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:14.067675 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:14.135571 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:14.156134 29378 consensus_peers.cc:485] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:14.163090 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:14.216178 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:14.218564 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:14.233812 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:14.234263 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:14.250365 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:14.253245 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:17:14.291949 29155 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:17:14.292994 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:14.305805 29421 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:17:14.314826 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:17:14.367936 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:17:14.426607 26538 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:17:14.447867 26406 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:17:14.452092 26798 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240502 14:17:14.452160 29378 consensus_peers.cc:485] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:14.490293 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:14.522246 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:14.580778 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:14.622155 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:14.626281 29378 consensus_peers.cc:485] T a511c4f2b4c14f4a90872bf0093c035c P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:14.699270 29378 consensus_peers.cc:485] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:14.764707 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:14.798619 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:14.873428 29378 consensus_peers.cc:485] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:14.892095 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:17:14.935622 29155 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:17:14.957029 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:14.993794 29421 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:17:15.015133 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:17:15.051812 26538 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:17:15.074508 26406 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:17:15.089509 26798 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240502 14:17:15.133925 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:15.198661 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:15.263762 26426 consensus_queue.cc:237] T 1fb1ef75829641a89c4cac539f431d80 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } }
I20240502 14:17:15.265936 29175 raft_consensus.cc:1261] T 1fb1ef75829641a89c4cac539f431d80 P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 14:17:15.265985 29575 raft_consensus.cc:1261] T 1fb1ef75829641a89c4cac539f431d80 P 4092902b3e674857a0147f1b4f2cdb86 [term 2 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 14:17:15.266713 13301 consensus_queue.cc:1035] T 1fb1ef75829641a89c4cac539f431d80 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:17:15.267046 13922 consensus_queue.cc:1035] T 1fb1ef75829641a89c4cac539f431d80 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:17:15.269946 13301 raft_consensus.cc:2932] T 1fb1ef75829641a89c4cac539f431d80 P 89af7f423cf44717a08d6b142f0ed49d [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:17:15.270375 29575 raft_consensus.cc:2932] T 1fb1ef75829641a89c4cac539f431d80 P 4092902b3e674857a0147f1b4f2cdb86 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:17:15.270582 29175 raft_consensus.cc:2932] T 1fb1ef75829641a89c4cac539f431d80 P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:17:15.277371 26283 catalog_manager.cc:5484] T 1fb1ef75829641a89c4cac539f431d80 P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: config changed from index 6 to 8, VOTER 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195) evicted. New cstate: current_term: 2 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } }
I20240502 14:17:15.289981 26669 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fb1ef75829641a89c4cac539f431d80 with delete_type TABLET_DATA_TOMBSTONED (TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:34150
I20240502 14:17:15.296969 14150 tablet_replica.cc:330] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: stopping tablet replica
I20240502 14:17:15.297331 14150 raft_consensus.cc:2229] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:17:15.297544 14150 raft_consensus.cc:2258] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:17:15.298432 14150 ts_tablet_manager.cc:1875] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 14:17:15.304365 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:15.313194 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:17:15.315140 14150 ts_tablet_manager.cc:1888] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240502 14:17:15.315423 14150 log.cc:1196] T 1fb1ef75829641a89c4cac539f431d80 P 69ab78b2a1b74d2bbf2cdeafa0a4af23: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-2/wal/wals/1fb1ef75829641a89c4cac539f431d80
I20240502 14:17:15.316520 26272 catalog_manager.cc:4830] TS 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325): tablet 1fb1ef75829641a89c4cac539f431d80 (table rebalance_test_table_13 [id=835ba73992f74008a1a10f4c18f2db38]) successfully deleted
W20240502 14:17:15.401073 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:15.503516 26624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.194:36025, user_credentials={real_user=slave}} blocked reactor thread for 70210.1us
W20240502 14:17:15.663643 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:15.707842 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:15.711423 29155 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:17:15.765070 29421 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:17:15.765764 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:15.769402 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:17:15.790534 26406 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:17:15.813340 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:17:15.834920 26538 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:17:15.845657 26798 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240502 14:17:15.904333 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:15.936636 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:16.031236 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:16.116572 29378 consensus_peers.cc:485] T cf808a060536445495a70de638438f87 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:16.143224 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:16.146828 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:16.235059 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:16.256114 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:16.266114 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:16.273195 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:16.274137 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:16.326258 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:16.453158 29378 consensus_peers.cc:485] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:17:16.453737 29155 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240502 14:17:16.493208 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:17:16.494057 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:16.512579 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:17:16.525575 29421 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:17:16.568812 29378 consensus_peers.cc:485] T c32ea761f2834d758529890e5b496f88 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:17:16.615309 26538 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240502 14:17:16.635210 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:16.636448 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:16.643282 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:16.650790 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:17:16.653766 26406 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:16.685074 29378 consensus_peers.cc:485] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:17:16.686870 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240502 14:17:16.703363 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:17:16.721822 26798 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240502 14:17:16.723285 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:16.747190 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:16.758773 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:16.849839 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:16.903108 26493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 126 similar messages]
W20240502 14:17:16.933593 29378 consensus_peers.cc:485] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:16.953022 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:16.962730 26754 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 14:17:16.963953 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:16.993148 26362 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 14:17:16.998481 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:17.025164 29378 consensus_peers.cc:485] T a511c4f2b4c14f4a90872bf0093c035c P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:17.052063 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:17.091995 29378 consensus_peers.cc:485] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:17.099717 26624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 14:17:17.102880 29155 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240502 14:17:17.118402 29111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 48 similar messages]
I20240502 14:17:17.165081 29421 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:17:17.165750 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:17.194525 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:17.200464 29378 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 14:17:17.201982 29378 consensus_peers.cc:485] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:17:17.244895 26406 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:17.252434 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:17.260637 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:17.262566 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:17.284453 26798 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:17:17.290442 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:17:17.299952 26538 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240502 14:17:17.382998 29510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:17:17.435956 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:17.650272 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:17.701344 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:17.729123 29155 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240502 14:17:17.730818 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:17:17.788301 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:17.819121 29421 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:17:17.838109 26406 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:17:17.838694 26538 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:17:17.863600 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:17:17.896471 26798 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240502 14:17:17.936865 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:18.012626 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:18.061209 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:18.164561 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:18.231132 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:18.253285 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:17:18.403769 29155 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:17:18.442902 26406 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:17:18.483573 29421 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:17:18.499061 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:17:18.500022 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:18.541563 26538 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:17:18.569108 26798 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240502 14:17:18.631769 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:18.660544 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:18.682543 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:18.683207 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:17:18.719357 29440 consensus_queue.cc:237] T ce03ee354c54457487ea939301ccd90e P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:17:18.721487 26686 raft_consensus.cc:1261] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 2 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 14:17:18.721484 29175 raft_consensus.cc:1261] T ce03ee354c54457487ea939301ccd90e P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 14:17:18.722033 14304 consensus_queue.cc:1035] T ce03ee354c54457487ea939301ccd90e P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:17:18.722353 13855 consensus_queue.cc:1035] T ce03ee354c54457487ea939301ccd90e P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:17:18.725582 14519 raft_consensus.cc:2932] T ce03ee354c54457487ea939301ccd90e P 68a3c2b17ed94e06aba66ed4178132ab [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:17:18.726212 29175 raft_consensus.cc:2932] T ce03ee354c54457487ea939301ccd90e P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:17:18.727165 26686 raft_consensus.cc:2932] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:17:18.736667 26284 catalog_manager.cc:5484] T ce03ee354c54457487ea939301ccd90e P 68a3c2b17ed94e06aba66ed4178132ab reported cstate change: config changed from index 6 to 8, VOTER e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196) evicted. New cstate: current_term: 2 leader_uuid: "68a3c2b17ed94e06aba66ed4178132ab" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
W20240502 14:17:18.750411 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:18.753710 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:18.775219 29378 consensus_peers.cc:485] T cf808a060536445495a70de638438f87 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:17:18.781471 26798 tablet_service.cc:1514] Processing DeleteTablet for tablet ce03ee354c54457487ea939301ccd90e with delete_type TABLET_DATA_TOMBSTONED (TS e4cfb8fc5a474177af6848a6efa13a27 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:56010
I20240502 14:17:18.796813 14528 tablet_replica.cc:330] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27: stopping tablet replica
I20240502 14:17:18.797238 14528 raft_consensus.cc:2229] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:17:18.797565 14528 raft_consensus.cc:2258] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:17:18.798436 14528 ts_tablet_manager.cc:1875] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 14:17:18.814042 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:18.819731 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:17:18.823709 14528 ts_tablet_manager.cc:1888] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240502 14:17:18.823912 14528 log.cc:1196] T ce03ee354c54457487ea939301ccd90e P e4cfb8fc5a474177af6848a6efa13a27: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-3/wal/wals/ce03ee354c54457487ea939301ccd90e
I20240502 14:17:18.824821 26272 catalog_manager.cc:4830] TS e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843): tablet ce03ee354c54457487ea939301ccd90e (table rebalance_test_table_17 [id=796786e80d834485b2cee5a25161aa97]) successfully deleted
W20240502 14:17:18.899477 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:18.923564 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:18.942313 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:18.993598 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:19.061100 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:19.079763 29378 consensus_peers.cc:485] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:17:19.084735 29155 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:17:19.117381 29421 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:17:19.152930 26406 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:17:19.161221 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:19.162675 29378 consensus_peers.cc:485] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:19.175185 29378 consensus_peers.cc:485] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:19.186290 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:17:19.193945 26538 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240502 14:17:19.226387 29378 consensus_peers.cc:485] T c32ea761f2834d758529890e5b496f88 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:19.253288 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:17:19.291452 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240502 14:17:19.302155 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:19.312276 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:19.312482 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:17:19.329608 26798 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240502 14:17:19.374181 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:19.412143 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:19.416194 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:19.441061 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:19.466405 29378 consensus_peers.cc:485] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:19.532042 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:19.551802 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:19.633965 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:19.684239 29378 consensus_peers.cc:485] T a511c4f2b4c14f4a90872bf0093c035c P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:19.771737 29378 consensus_peers.cc:485] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:19.795300 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:19.818650 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:19.840211 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:19.842636 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:19.845105 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:19.878597 29421 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:17:19.879458 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:19.923566 29155 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:17:19.932458 26538 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:17:19.973852 26406 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:17:19.986431 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:17:19.990128 26798 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240502 14:17:20.069635 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:20.286633 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:20.306357 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:20.389809 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:20.525470 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:17:20.635835 29155 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:17:20.640280 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:20.667454 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:17:20.679527 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:17:20.679976 26406 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:17:20.684677 29421 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:17:20.693331 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:17:20.732820 26538 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:17:20.800648 26798 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240502 14:17:20.899541 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:20.954298 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:21.119172 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:21.205853 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:21.215785 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:21.234254 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:21.268875 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:17:21.269991 29155 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:17:21.292223 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:21.302017 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:17:21.303120 29421 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:17:21.338335 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:21.365145 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:17:21.378760 26406 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:21.379542 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:21.415758 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:17:21.429826 26538 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240502 14:17:21.475673 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:21.499696 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:21.500778 26798 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240502 14:17:21.502213 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:21.503214 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:21.568770 29378 consensus_peers.cc:485] T b8430c13474b4850b71020c4a89bea68 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:21.585626 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:17:21.625742 29440 consensus_queue.cc:237] T 9f3cbedfa1da42698615a544d9dc3276 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:17:21.628547 29174 raft_consensus.cc:1261] T 9f3cbedfa1da42698615a544d9dc3276 P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 14:17:21.628691 26558 raft_consensus.cc:1261] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a [term 2 FOLLOWER]: Refusing update from remote peer 68a3c2b17ed94e06aba66ed4178132ab: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 14:17:21.629115 14304 consensus_queue.cc:1035] T 9f3cbedfa1da42698615a544d9dc3276 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:17:21.629432 14717 consensus_queue.cc:1035] T 9f3cbedfa1da42698615a544d9dc3276 P 68a3c2b17ed94e06aba66ed4178132ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:17:21.633242 14717 raft_consensus.cc:2932] T 9f3cbedfa1da42698615a544d9dc3276 P 68a3c2b17ed94e06aba66ed4178132ab [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:17:21.633913 29174 raft_consensus.cc:2932] T 9f3cbedfa1da42698615a544d9dc3276 P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:17:21.642800 26276 catalog_manager.cc:5484] T 9f3cbedfa1da42698615a544d9dc3276 P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: config changed from index 6 to 8, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) evicted. New cstate: current_term: 2 leader_uuid: "68a3c2b17ed94e06aba66ed4178132ab" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:17:21.645164 26558 raft_consensus.cc:2932] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:17:21.662670 26406 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f3cbedfa1da42698615a544d9dc3276 with delete_type TABLET_DATA_TOMBSTONED (TS 89af7f423cf44717a08d6b142f0ed49d not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:37422
I20240502 14:17:21.664799 14824 tablet_replica.cc:330] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d: stopping tablet replica
I20240502 14:17:21.665032 14824 raft_consensus.cc:2229] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:17:21.665242 14824 raft_consensus.cc:2258] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:17:21.666437 14824 ts_tablet_manager.cc:1875] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 14:17:21.677601 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:21.685231 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:17:21.688378 14824 ts_tablet_manager.cc:1888] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240502 14:17:21.688588 14824 log.cc:1196] T 9f3cbedfa1da42698615a544d9dc3276 P 89af7f423cf44717a08d6b142f0ed49d: Deleting WAL directory at /tmp/dist-test-task_zeJNl/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714659195971712-19835-0/raft_consensus-itest-cluster/ts-0/wal/wals/9f3cbedfa1da42698615a544d9dc3276
I20240502 14:17:21.689414 26271 catalog_manager.cc:4830] TS 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985): tablet 9f3cbedfa1da42698615a544d9dc3276 (table rebalance_test_table_7 [id=998b08830b5f4b6fbb49d494f21739b9]) successfully deleted
W20240502 14:17:21.689478 29378 consensus_peers.cc:485] T 4cea6495041642f58e1eb5d26fc10df5 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:21.717497 29378 consensus_peers.cc:485] T cf808a060536445495a70de638438f87 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:21.734222 29378 consensus_peers.cc:485] T c32ea761f2834d758529890e5b496f88 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:17:21.750681 26819 consensus_queue.cc:237] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:17:21.753914 29440 raft_consensus.cc:1261] T 533b42de2de14abc96c00a928d5deed1 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:17:21.754305 26425 raft_consensus.cc:1261] T 533b42de2de14abc96c00a928d5deed1 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 14:17:21.755059 26754 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715): Couldn't send request to peer 1cb67158133e4648a6eceba32d4dbba3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 533b42de2de14abc96c00a928d5deed1. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:17:21.755080 14688 consensus_queue.cc:1035] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:21.755760 14688 consensus_queue.cc:1035] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:21.770287 14249 raft_consensus.cc:2932] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:17:21.771155 29440 raft_consensus.cc:2932] T 533b42de2de14abc96c00a928d5deed1 P 68a3c2b17ed94e06aba66ed4178132ab [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:17:21.774586 26425 raft_consensus.cc:2932] T 533b42de2de14abc96c00a928d5deed1 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
I20240502 14:17:21.777088 26284 catalog_manager.cc:5484] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: config changed from index 5 to 6, VOTER 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197) added. New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
W20240502 14:17:21.788888 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:17:21.830751 26819 consensus_queue.cc:237] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
W20240502 14:17:21.830921 29378 consensus_peers.cc:485] T a511c4f2b4c14f4a90872bf0093c035c P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:17:21.833267 26425 raft_consensus.cc:1261] T 39a61744642f40c8ae92da708dc91ca3 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:17:21.833453 29174 raft_consensus.cc:1261] T 39a61744642f40c8ae92da708dc91ca3 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:17:21.834223 14249 consensus_queue.cc:1035] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:17:21.834532 14688 consensus_queue.cc:1035] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 14:17:21.835387 26755 consensus_peers.cc:485] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39a61744642f40c8ae92da708dc91ca3. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:17:21.838459 14249 raft_consensus.cc:2932] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:17:21.840066 26425 raft_consensus.cc:2932] T 39a61744642f40c8ae92da708dc91ca3 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:17:21.844480 29174 raft_consensus.cc:2932] T 39a61744642f40c8ae92da708dc91ca3 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:17:21.848966 26285 catalog_manager.cc:5484] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: config changed from index 5 to 6, VOTER 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) added. New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
W20240502 14:17:21.856812 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:21.876960 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:21.899818 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:17:21.900631 26818 consensus_queue.cc:237] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:17:21.903919 29573 raft_consensus.cc:1261] T dc3740d0671d42bc90eb9baeb02f5b0a P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:17:21.904997 26689 raft_consensus.cc:1261] T dc3740d0671d42bc90eb9baeb02f5b0a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:17:21.905185 13699 consensus_queue.cc:1035] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:17:21.907001 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:17:21.907567 13699 consensus_queue.cc:1035] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:17:21.911149 26493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 128 similar messages]
I20240502 14:17:21.911716 14688 raft_consensus.cc:2932] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
W20240502 14:17:21.913538 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:17:21.914273 26689 raft_consensus.cc:2932] T dc3740d0671d42bc90eb9baeb02f5b0a P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:17:21.923267 26285 catalog_manager.cc:5484] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: config changed from index 3 to 4, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New cstate: current_term: 1 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
I20240502 14:17:21.925954 29573 raft_consensus.cc:2932] T dc3740d0671d42bc90eb9baeb02f5b0a P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } }
W20240502 14:17:21.930440 29378 consensus_peers.cc:485] T 457516f59dee4436b474af63819fad4e P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:21.947253 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:21.977592 26754 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 14:17:22.002836 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:22.008370 26362 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 14:17:22.031270 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:22.072626 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:17:22.165697 14931 ts_tablet_manager.cc:902] T 533b42de2de14abc96c00a928d5deed1 P 1cb67158133e4648a6eceba32d4dbba3: Initiating tablet copy from peer e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:17:22.169682 14931 tablet_copy_client.cc:308] T 533b42de2de14abc96c00a928d5deed1 P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Beginning tablet copy session from remote peer at address 127.19.94.196:32843
I20240502 14:17:22.172890 26839 tablet_copy_service.cc:140] P e4cfb8fc5a474177af6848a6efa13a27: Received BeginTabletCopySession request for tablet 533b42de2de14abc96c00a928d5deed1 from peer 1cb67158133e4648a6eceba32d4dbba3 ({username='slave'} at 127.19.94.197:46803)
I20240502 14:17:22.173092 26839 tablet_copy_service.cc:161] P e4cfb8fc5a474177af6848a6efa13a27: Beginning new tablet copy session on tablet 533b42de2de14abc96c00a928d5deed1 from peer 1cb67158133e4648a6eceba32d4dbba3 at {username='slave'} at 127.19.94.197:46803: session id = 1cb67158133e4648a6eceba32d4dbba3-533b42de2de14abc96c00a928d5deed1
I20240502 14:17:22.175516 26839 tablet_copy_source_session.cc:205] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:17:22.177148 14931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 533b42de2de14abc96c00a928d5deed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:17:22.183768 14931 tablet_copy_client.cc:783] T 533b42de2de14abc96c00a928d5deed1 P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Starting download of 5 data blocks...
I20240502 14:17:22.199250 14931 tablet_copy_client.cc:647] T 533b42de2de14abc96c00a928d5deed1 P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Starting download of 1 WAL segments...
I20240502 14:17:22.201694 14931 tablet_copy_client.cc:523] T 533b42de2de14abc96c00a928d5deed1 P 1cb67158133e4648a6eceba32d4dbba3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240502 14:17:22.206650 29111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 14:17:22.207710 29378 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 76 similar messages]
I20240502 14:17:22.211445 14931 tablet_bootstrap.cc:492] T 533b42de2de14abc96c00a928d5deed1 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap starting.
W20240502 14:17:22.211730 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:17:22.224486 29155 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:17:22.269822 14943 ts_tablet_manager.cc:902] T 39a61744642f40c8ae92da708dc91ca3 P 68a3c2b17ed94e06aba66ed4178132ab: Initiating tablet copy from peer e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
F20240502 14:17:22.270181 14943 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-19f81d8b28564c0dbb0b80bd5018ed5a.queue_time_us, tablet-download-pool-19f81d8b28564c0dbb0b80bd5018ed5a.queue_time_us) (tablet-download-pool-19f81d8b28564c0dbb0b80bd5018ed5a.run_cpu_time_us, tablet-download-pool-19f81d8b28564c0dbb0b80bd5018ed5a.run_cpu_time_us) (tablet-download-pool-19f81d8b28564c0dbb0b80bd5018ed5a.run_wall_time_us, tablet-download-pool-19f81d8b28564c0dbb0b80bd5018ed5a.run_wall_time_us) (tablet-download-pool-2108a04310804945b368bcb5f88170e2.queue_time_us, tablet-download-pool-2108a04310804945b368bcb5f88170e2.queue_time_us) (tablet-download-pool-2108a04310804945b368bcb5f88170e2.run_cpu_time_us, tablet-download-pool-2108a04310804945b368bcb5f88170e2.run_cpu_time_us) (tablet-download-pool-2108a04310804945b368bcb5f88170e2.run_wall_time_us, tablet-download-pool-2108a04310804945b368bcb5f88170e2.run_wall_time_us) (tablet-download-pool-2dc3b47b4ab145ccb9efbf1b5740c7f6.queue_time_us, tablet-download-pool-2dc3b47b4ab145ccb9efbf1b5740c7f6.queue_time_us) (tablet-download-pool-2dc3b47b4ab145ccb9efbf1b5740c7f6.run_cpu_time_us, tablet-download-pool-2dc3b47b4ab145ccb9efbf1b5740c7f6.run_cpu_time_us) (tablet-download-pool-2dc3b47b4ab145ccb9efbf1b5740c7f6.run_wall_time_us, tablet-download-pool-2dc3b47b4ab145ccb9efbf1b5740c7f6.run_wall_time_us) (tablet-download-pool-39a61744642f40c8ae92da708dc91ca3.queue_time_us, tablet-download-pool-39a61744642f40c8ae92da708dc91ca3.queue_time_us) (tablet-download-pool-451b303a32a3404d8d584203d6704b35.queue_time_us, tablet-download-pool-451b303a32a3404d8d584203d6704b35.queue_time_us) (tablet-download-pool-451b303a32a3404d8d584203d6704b35.run_cpu_time_us, tablet-download-pool-451b303a32a3404d8d584203d6704b35.run_cpu_time_us) (tablet-download-pool-451b303a32a3404d8d584203d6704b35.run_wall_time_us, tablet-download-pool-451b303a32a3404d8d584203d6704b35.run_wall_time_us) (tablet-download-pool-4bf6e7410afa4c048f5842ec2a426d77.queue_time_us, tablet-download-pool-4bf6e7410afa4c048f5842ec2a426d77.queue_time_us) (tablet-download-pool-4bf6e7410afa4c048f5842ec2a426d77.run_cpu_time_us, tablet-download-pool-4bf6e7410afa4c048f5842ec2a426d77.run_cpu_time_us) (tablet-download-pool-4bf6e7410afa4c048f5842ec2a426d77.run_wall_time_us, tablet-download-pool-4bf6e7410afa4c048f5842ec2a426d77.run_wall_time_us) (tablet-download-pool-4df2bd4e182c4fe991da385d92d3f76f.queue_time_us, tablet-download-pool-4df2bd4e182c4fe991da385d92d3f76f.queue_time_us) (tablet-download-pool-4df2bd4e182c4fe991da385d92d3f76f.run_cpu_time_us, tablet-download-pool-4df2bd4e182c4fe991da385d92d3f76f.run_cpu_time_us) (tablet-download-pool-4df2bd4e182c4fe991da385d92d3f76f.run_wall_time_us, tablet-download-pool-4df2bd4e182c4fe991da385d92d3f76f.run_wall_time_us) (tablet-download-pool-533b42de2de14abc96c00a928d5deed1.queue_time_us, tablet-download-pool-533b42de2de14abc96c00a928d5deed1.queue_time_us) (tablet-download-pool-533b42de2de14abc96c00a928d5deed1.run_cpu_time_us, tablet-download-pool-533b42de2de14abc96c00a928d5deed1.run_cpu_time_us) (tablet-download-pool-533b42de2de14abc96c00a928d5deed1.run_wall_time_us, tablet-download-pool-533b42de2de14abc96c00a928d5deed1.run_wall_time_us) (tablet-download-pool-736ed8faceec4f0fa07c62952ea32bf6.queue_time_us, tablet-download-pool-736ed8faceec4f0fa07c62952ea32bf6.queue_time_us) (tablet-download-pool-736ed8faceec4f0fa07c62952ea32bf6.run_cpu_time_us, tablet-download-pool-736ed8faceec4f0fa07c62952ea32bf6.run_cpu_time_us) (tablet-download-pool-736ed8faceec4f0fa07c62952ea32bf6.run_wall_time_us, tablet-download-pool-736ed8faceec4f0fa07c62952ea32bf6.run_wall_time_us) (tablet-download-pool-9f3cbedfa1da42698615a544d9dc3276.queue_time_us, tablet-download-pool-9f3cbedfa1da42698615a544d9dc3276.queue_time_us) (tablet-download-pool-9f3cbedfa1da42698615a544d9dc3276.run_cpu_time_us, tablet-download-pool-9f3cbedfa1da42698615a544d9dc3276.run_cpu_time_us) (tablet-download-pool-9f3cbedfa1da42698615a544d9dc3276.run_wall_time_us, tablet-download-pool-9f3cbedfa1da42698615a544d9dc3276.run_wall_time_us) (tablet-download-pool-a42e68ab03f84d4b92893006fcf6f229.queue_time_us, tablet-download-pool-a42e68ab03f84d4b92893006fcf6f229.queue_time_us) (tablet-download-pool-a42e68ab03f84d4b92893006fcf6f229.run_cpu_time_us, tablet-download-pool-a42e68ab03f84d4b92893006fcf6f229.run_cpu_time_us) (tablet-download-pool-a42e68ab03f84d4b92893006fcf6f229.run_wall_time_us, tablet-download-pool-a42e68ab03f84d4b92893006fcf6f229.run_wall_time_us) (tablet-download-pool-c32ea761f2834d758529890e5b496f88.queue_time_us, tablet-download-pool-c32ea761f2834d758529890e5b496f88.queue_time_us) (tablet-download-pool-c32ea761f2834d758529890e5b496f88.run_cpu_time_us, tablet-download-pool-c32ea761f2834d758529890e5b496f88.run_cpu_time_us) (tablet-download-pool-c32ea761f2834d758529890e5b496f88.run_wall_time_us, tablet-download-pool-c32ea761f2834d758529890e5b496f88.run_wall_time_us) (tablet-download-pool-c41522c60f3942aa9f28f514dcad4e49.queue_time_us, tablet-download-pool-c41522c60f3942aa9f28f514dcad4e49.queue_time_us) (tablet-download-pool-c41522c60f3942aa9f28f514dcad4e49.run_cpu_time_us, tablet-download-pool-c41522c60f3942aa9f28f514dcad4e49.run_cpu_time_us) (tablet-download-pool-c41522c60f3942aa9f28f514dcad4e49.run_wall_time_us, tablet-download-pool-c41522c60f3942aa9f28f514dcad4e49.run_wall_time_us) (tablet-download-pool-ce03ee354c54457487ea939301ccd90e.queue_time_us, tablet-download-pool-ce03ee354c54457487ea939301ccd90e.queue_time_us) (tablet-download-pool-ce03ee354c54457487ea939301ccd90e.run_cpu_time_us, tablet-download-pool-ce03ee354c54457487ea939301ccd90e.run_cpu_time_us) (tablet-download-pool-ce03ee354c54457487ea939301ccd90e.run_wall_time_us, tablet-download-pool-ce03ee354c54457487ea939301ccd90e.run_wall_time_us) (tablet-download-pool-cf52d0f868fa45418af48639e12f3282.queue_time_us, tablet-download-pool-cf52d0f868fa45418af48639e12f3282.queue_time_us) (tablet-download-pool-cf52d0f868fa45418af48639e12f3282.run_cpu_time_us, tablet-download-pool-cf52d0f868fa45418af48639e12f3282.run_cpu_time_us) (tablet-download-pool-cf52d0f868fa45418af48639e12f3282.run_wall_time_us, tablet-download-pool-cf52d0f868fa45418af48639e12f3282.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240502 14:17:22.278079 14931 tablet_bootstrap.cc:492] T 533b42de2de14abc96c00a928d5deed1 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:17:22.279037 14931 tablet_bootstrap.cc:492] T 533b42de2de14abc96c00a928d5deed1 P 1cb67158133e4648a6eceba32d4dbba3: Bootstrap complete.
I20240502 14:17:22.279932 14931 ts_tablet_manager.cc:1367] T 533b42de2de14abc96c00a928d5deed1 P 1cb67158133e4648a6eceba32d4dbba3: Time spent bootstrapping tablet: real 0.069s user 0.021s sys 0.000s
I20240502 14:17:22.280828 14931 raft_consensus.cc:348] T 533b42de2de14abc96c00a928d5deed1 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:17:22.281075 14931 raft_consensus.cc:729] T 533b42de2de14abc96c00a928d5deed1 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Initialized, Role: FOLLOWER
I20240502 14:17:22.281313 14931 consensus_queue.cc:260] T 533b42de2de14abc96c00a928d5deed1 P 1cb67158133e4648a6eceba32d4dbba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
@ 0x7fa95efb6dcd google::LogMessage::Fail() at ??:0
I20240502 14:17:22.285874 14931 ts_tablet_manager.cc:1398] T 533b42de2de14abc96c00a928d5deed1 P 1cb67158133e4648a6eceba32d4dbba3: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
@ 0x7fa95efbab93 google::LogMessage::SendToLog() at ??:0
W20240502 14:17:22.294772 29378 consensus_peers.cc:485] T e93acdb9427943938f4c12ae2bf2c594 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
@ 0x7fa95efb67cc google::LogMessage::Flush() at ??:0
I20240502 14:17:22.295776 26839 tablet_copy_service.cc:342] P e4cfb8fc5a474177af6848a6efa13a27: Request end of tablet copy session 1cb67158133e4648a6eceba32d4dbba3-533b42de2de14abc96c00a928d5deed1 received from {username='slave'} at 127.19.94.197:46803
I20240502 14:17:22.295961 26839 tablet_copy_service.cc:433] P e4cfb8fc5a474177af6848a6efa13a27: ending tablet copy session 1cb67158133e4648a6eceba32d4dbba3-533b42de2de14abc96c00a928d5deed1 on tablet 533b42de2de14abc96c00a928d5deed1 with peer 1cb67158133e4648a6eceba32d4dbba3
@ 0x7fa95efb7f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
W20240502 14:17:22.303406 26624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 41 similar messages]
@ 0x7fa961cf7742 kudu::TraceMetrics::InternName() at ??:0
W20240502 14:17:22.312326 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:17:22.320499 29222 maintenance_manager.cc:419] P 1cb67158133e4648a6eceba32d4dbba3: Scheduling UndoDeltaBlockGCOp(533b42de2de14abc96c00a928d5deed1): 4838 bytes on disk
I20240502 14:17:22.321522 29113 maintenance_manager.cc:643] P 1cb67158133e4648a6eceba32d4dbba3: UndoDeltaBlockGCOp(533b42de2de14abc96c00a928d5deed1) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":130,"lbm_reads_lt_1ms":4}
@ 0x7fa961cc3f8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7fa961cbdcd6 kudu::ThreadPoolBuilder::Build() at ??:0
W20240502 14:17:22.343022 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:17:22.347793 29421 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:17:22.354856 29378 consensus_peers.cc:485] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P 68a3c2b17ed94e06aba66ed4178132ab -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
@ 0x7fa9711108de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240502 14:17:22.360304 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
@ 0x7fa971128daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7fa971253480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7fa97128ba8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240502 14:17:22.381035 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:22.385380 29111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.193:37985, user_credentials={real_user=slave}} blocked reactor thread for 53670.4us
@ 0x7fa97126e981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7fa97126e70e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
W20240502 14:17:22.482336 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:17:22.484819 26538 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:17:22.500133 26798 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
@ 0x7fa961cc7b75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa961ccb19c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fa961ccb019 std::_Function_handler<>::_M_invoke() at ??:0
W20240502 14:17:22.533528 29510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 20 similar messages]
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240502 14:17:22.543879 26406 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:22.554123 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
@ 0x7fa961ca601e kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa96502b6db start_thread at ??:0
@ 0x7fa95de6971f clone at ??:0
W20240502 14:17:22.655637 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:17:22.705559 29175 raft_consensus.cc:1203] T 533b42de2de14abc96c00a928d5deed1 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
W20240502 14:17:22.866082 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:22.926770 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:22.973503 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:22.986814 29112 connection.cc:535] client connection to 127.19.94.199:41781 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:17:22.987174 26625 connection.cc:535] client connection to 127.19.94.199:41781 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:17:22.987432 26494 connection.cc:535] client connection to 127.19.94.199:41781 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:17:22.988673 29511 connection.cc:535] client connection to 127.19.94.199:41781 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:17:22.989012 26363 connection.cc:535] client connection to 127.19.94.199:41781 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:17:22.990033 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:17:22.990144 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:17:22.990221 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:17:22.990310 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:17:22.990391 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:17:22.990482 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:17:22.990558 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:17:22.990628 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:17:22.990698 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:17:22.990774 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:17:22.991317 12236 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:17:22.988270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.94.199:41781 (local address 127.19.94.195:50625)
0502 14:17:22.988430 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0502 14:17:22.988448 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0502 14:17:22.988580 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:17:22.991153 (+ 2573us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":32}
W20240502 14:17:22.991503 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:17:22.991619 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:17:22.991701 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:17:22.991765 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:17:22.991825 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:17:22.993985 12247 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:17:22.988178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.94.199:41781 (local address 127.19.94.197:37793)
0502 14:17:22.988558 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0502 14:17:22.988579 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0502 14:17:22.988729 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:17:22.993251 (+ 4522us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":194}
W20240502 14:17:22.994243 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:17:22.994349 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:17:22.994793 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:17:22.994971 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:17:22.995213 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:17:22.995363 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:17:22.995823 26755 connection.cc:535] client connection to 127.19.94.199:41781 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:17:22.998967 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:17:22.999146 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:22.999246 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:17:22.999367 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:17:22.999477 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:17:22.999692 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:17:22.999912 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:17:23.004246 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:23.025339 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:17:23.118798 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:17:23.128779 29155 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:17:23.145539 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:23.148061 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240502 14:17:23.237839 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:17:23.241606 26798 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:17:23.251299 26406 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:17:23.260622 26538 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240502 14:17:23.294325 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:23.296871 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:23.401876 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:23.534721 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:23.552523 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:17:23.561226 14688 raft_consensus.cc:482] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:23.561517 14688 raft_consensus.cc:504] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:17:23.562427 14688 leader_election.cc:288] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:17:23.568292 26755 leader_election.cc:334] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:23.570950 26754 leader_election.cc:334] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:23.571105 26754 leader_election.cc:302] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:23.572335 14688 raft_consensus.cc:2726] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:17:23.572422 14949 raft_consensus.cc:482] T 01e6bc3689db46e2ad7d0484a29af287 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:23.572553 14949 raft_consensus.cc:504] T 01e6bc3689db46e2ad7d0484a29af287 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:17:23.573362 14949 leader_election.cc:288] T 01e6bc3689db46e2ad7d0484a29af287 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:17:23.573817 26556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01e6bc3689db46e2ad7d0484a29af287" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
W20240502 14:17:23.576937 29112 leader_election.cc:334] T 01e6bc3689db46e2ad7d0484a29af287 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:23.577113 29112 leader_election.cc:302] T 01e6bc3689db46e2ad7d0484a29af287 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3; no voters: 68a3c2b17ed94e06aba66ed4178132ab, ba187e38a2754d60a12de9ac4856430a
I20240502 14:17:23.577695 14949 raft_consensus.cc:2726] T 01e6bc3689db46e2ad7d0484a29af287 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:17:23.579233 14892 raft_consensus.cc:482] T 177a34c96aa84cb4aa50bbd79cab4144 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:23.579422 14892 raft_consensus.cc:504] T 177a34c96aa84cb4aa50bbd79cab4144 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:17:23.580129 14892 leader_election.cc:288] T 177a34c96aa84cb4aa50bbd79cab4144 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:17:23.580837 29175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "177a34c96aa84cb4aa50bbd79cab4144" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:17:23.585817 14892 raft_consensus.cc:482] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:23.585986 14892 raft_consensus.cc:504] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:17:23.586624 14892 leader_election.cc:288] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
W20240502 14:17:23.591063 26494 leader_election.cc:334] T 177a34c96aa84cb4aa50bbd79cab4144 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:23.591133 26493 leader_election.cc:334] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:23.591251 26494 leader_election.cc:302] T 177a34c96aa84cb4aa50bbd79cab4144 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 1cb67158133e4648a6eceba32d4dbba3, 68a3c2b17ed94e06aba66ed4178132ab
I20240502 14:17:23.591714 14892 raft_consensus.cc:2726] T 177a34c96aa84cb4aa50bbd79cab4144 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:17:23.595594 26494 leader_election.cc:334] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:23.595736 26494 leader_election.cc:302] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:23.595978 14892 raft_consensus.cc:2726] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:17:23.607163 14949 raft_consensus.cc:482] T ce03ee354c54457487ea939301ccd90e P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:23.607367 14949 raft_consensus.cc:504] T ce03ee354c54457487ea939301ccd90e P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:17:23.609856 14949 leader_election.cc:288] T ce03ee354c54457487ea939301ccd90e P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195:44325), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:17:23.610786 26689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce03ee354c54457487ea939301ccd90e" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" is_pre_election: true
W20240502 14:17:23.613869 29112 leader_election.cc:334] T ce03ee354c54457487ea939301ccd90e P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:23.614164 29112 leader_election.cc:302] T ce03ee354c54457487ea939301ccd90e P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3; no voters: 68a3c2b17ed94e06aba66ed4178132ab, 69ab78b2a1b74d2bbf2cdeafa0a4af23
I20240502 14:17:23.614395 14949 raft_consensus.cc:2726] T ce03ee354c54457487ea939301ccd90e P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:17:23.676038 29155 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:17:23.682535 14362 raft_consensus.cc:482] T ffaded0bc6204e8c8576ea8eabdd18ab P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:23.682709 14362 raft_consensus.cc:504] T ffaded0bc6204e8c8576ea8eabdd18ab P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:17:23.683457 14362 leader_election.cc:288] T ffaded0bc6204e8c8576ea8eabdd18ab P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843)
I20240502 14:17:23.686193 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffaded0bc6204e8c8576ea8eabdd18ab" candidate_uuid: "4092902b3e674857a0147f1b4f2cdb86" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
W20240502 14:17:23.687172 29511 leader_election.cc:334] T ffaded0bc6204e8c8576ea8eabdd18ab P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:23.687539 29511 leader_election.cc:302] T ffaded0bc6204e8c8576ea8eabdd18ab P 4092902b3e674857a0147f1b4f2cdb86 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86; no voters: 68a3c2b17ed94e06aba66ed4178132ab, e4cfb8fc5a474177af6848a6efa13a27
I20240502 14:17:23.688107 14362 raft_consensus.cc:2726] T ffaded0bc6204e8c8576ea8eabdd18ab P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:17:23.705785 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:23.709749 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:23.709920 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:17:23.723157 14586 raft_consensus.cc:482] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:23.723358 14586 raft_consensus.cc:504] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:17:23.724128 14586 leader_election.cc:288] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:17:23.725988 14949 raft_consensus.cc:482] T eb065120d1304a2a81ce932934b911f3 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:23.726223 14949 raft_consensus.cc:504] T eb065120d1304a2a81ce932934b911f3 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
W20240502 14:17:23.726519 26493 leader_election.cc:334] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:23.727147 14949 leader_election.cc:288] T eb065120d1304a2a81ce932934b911f3 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985)
I20240502 14:17:23.735286 14949 raft_consensus.cc:482] T 9f3cbedfa1da42698615a544d9dc3276 P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:23.735479 14949 raft_consensus.cc:504] T 9f3cbedfa1da42698615a544d9dc3276 P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:17:23.736114 14949 leader_election.cc:288] T 9f3cbedfa1da42698615a544d9dc3276 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:17:23.737005 26556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f3cbedfa1da42698615a544d9dc3276" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
W20240502 14:17:23.739845 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:23.741053 29112 leader_election.cc:334] T eb065120d1304a2a81ce932934b911f3 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:23.741204 26424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb065120d1304a2a81ce932934b911f3" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
W20240502 14:17:23.741799 26494 leader_election.cc:334] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:23.741972 26494 leader_election.cc:302] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:23.742265 14892 raft_consensus.cc:2726] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:17:23.744124 29111 leader_election.cc:302] T eb065120d1304a2a81ce932934b911f3 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3; no voters: 68a3c2b17ed94e06aba66ed4178132ab, 89af7f423cf44717a08d6b142f0ed49d
I20240502 14:17:23.744313 26406 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:17:23.744495 26538 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:17:23.744835 14716 raft_consensus.cc:2726] T eb065120d1304a2a81ce932934b911f3 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:17:23.750187 29112 leader_election.cc:334] T 9f3cbedfa1da42698615a544d9dc3276 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:23.750380 29112 leader_election.cc:302] T 9f3cbedfa1da42698615a544d9dc3276 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3; no voters: 68a3c2b17ed94e06aba66ed4178132ab, ba187e38a2754d60a12de9ac4856430a
I20240502 14:17:23.751224 14716 raft_consensus.cc:2726] T 9f3cbedfa1da42698615a544d9dc3276 P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:17:23.771273 15100 raft_consensus.cc:482] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:23.773878 26669 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:17:23.779812 14662 raft_consensus.cc:482] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:23.780014 14662 raft_consensus.cc:504] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
W20240502 14:17:23.783401 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:17:23.772337 15100 raft_consensus.cc:504] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
W20240502 14:17:23.792845 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:23.801743 14892 raft_consensus.cc:482] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:23.801954 14892 raft_consensus.cc:504] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:17:23.802800 14892 leader_election.cc:288] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:17:23.800289 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:17:23.803426 29175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01e6bc3689db46e2ad7d0484a29af287" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:17:23.803700 29175 raft_consensus.cc:2454] T 01e6bc3689db46e2ad7d0484a29af287 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 1.
W20240502 14:17:23.804313 26494 leader_election.cc:334] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:23.804780 26493 leader_election.cc:302] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab
I20240502 14:17:23.805483 14892 raft_consensus.cc:2781] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:17:23.805621 14892 raft_consensus.cc:482] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:23.805756 14892 raft_consensus.cc:3037] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:17:23.806191 14662 leader_election.cc:288] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
W20240502 14:17:23.808720 26362 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:23.809127 14892 raft_consensus.cc:504] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:17:23.810616 29175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01e6bc3689db46e2ad7d0484a29af287" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:17:23.810876 29175 raft_consensus.cc:3037] T 01e6bc3689db46e2ad7d0484a29af287 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:17:23.814419 29175 raft_consensus.cc:2454] T 01e6bc3689db46e2ad7d0484a29af287 P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 2.
I20240502 14:17:23.816788 26493 leader_election.cc:302] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:17:23.817672 14586 raft_consensus.cc:2781] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:17:23.817919 14586 raft_consensus.cc:686] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a [term 2 LEADER]: Becoming Leader. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Running, Role: LEADER
I20240502 14:17:23.818435 14586 consensus_queue.cc:237] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:17:23.821542 15100 leader_election.cc:288] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
W20240502 14:17:23.822345 26494 leader_election.cc:334] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:23.824003 29112 leader_election.cc:334] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:23.817705 14892 leader_election.cc:288] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 election: Requested vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:17:23.838430 14892 raft_consensus.cc:482] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:23.838794 14892 raft_consensus.cc:504] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:17:23.839967 14892 leader_election.cc:288] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:17:23.846608 26493 leader_election.cc:334] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:23.848143 14688 raft_consensus.cc:482] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:23.848436 14688 raft_consensus.cc:504] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:17:23.849767 14688 leader_election.cc:288] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
W20240502 14:17:23.852363 26754 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:23.852802 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:17:23.853957 26425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cea6495041642f58e1eb5d26fc10df5" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:17:23.854195 26425 raft_consensus.cc:2454] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
W20240502 14:17:23.856978 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:23.859624 26494 leader_election.cc:334] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:23.859954 26494 leader_election.cc:302] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:23.861810 14586 raft_consensus.cc:2726] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:17:23.862432 14900 raft_consensus.cc:482] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:23.862604 14900 raft_consensus.cc:504] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:17:23.863291 14900 leader_election.cc:288] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:17:23.864873 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cea6495041642f58e1eb5d26fc10df5" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:17:23.865178 26818 raft_consensus.cc:2454] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
W20240502 14:17:23.865489 26362 leader_election.cc:334] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:23.869988 26285 catalog_manager.cc:5484] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a reported cstate change: term changed from 1 to 2, leader changed from 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) to ba187e38a2754d60a12de9ac4856430a (127.19.94.194). New cstate: current_term: 2 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
W20240502 14:17:23.870726 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:23.872428 26363 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:23.872606 26363 leader_election.cc:302] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:23.872844 14900 raft_consensus.cc:2726] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:17:23.874325 26755 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:23.874513 26755 leader_election.cc:302] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:23.875250 14885 raft_consensus.cc:2726] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:17:23.876922 26363 leader_election.cc:334] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:23.877060 26363 leader_election.cc:302] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:23.877403 14900 raft_consensus.cc:2726] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:17:23.884979 26556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "177a34c96aa84cb4aa50bbd79cab4144" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a" is_pre_election: true
I20240502 14:17:23.885250 26556 raft_consensus.cc:2454] T 177a34c96aa84cb4aa50bbd79cab4144 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 1.
I20240502 14:17:23.885967 29111 leader_election.cc:302] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab
I20240502 14:17:23.886778 14949 raft_consensus.cc:2781] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:17:23.886879 14949 raft_consensus.cc:482] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:23.886953 14949 raft_consensus.cc:3037] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:17:23.889194 14949 raft_consensus.cc:504] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
W20240502 14:17:23.891503 29112 leader_election.cc:334] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:23.892113 14949 leader_election.cc:288] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 2 election: Requested vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), ba187e38a2754d60a12de9ac4856430a (127.19.94.194:36025)
I20240502 14:17:23.892582 26558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "177a34c96aa84cb4aa50bbd79cab4144" candidate_uuid: "1cb67158133e4648a6eceba32d4dbba3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ba187e38a2754d60a12de9ac4856430a"
I20240502 14:17:23.892823 26558 raft_consensus.cc:3037] T 177a34c96aa84cb4aa50bbd79cab4144 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:17:23.895251 26558 raft_consensus.cc:2454] T 177a34c96aa84cb4aa50bbd79cab4144 P ba187e38a2754d60a12de9ac4856430a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb67158133e4648a6eceba32d4dbba3 in term 2.
I20240502 14:17:23.896282 29111 leader_election.cc:302] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab
I20240502 14:17:23.896672 14949 raft_consensus.cc:2781] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:17:23.896831 14949 raft_consensus.cc:686] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 [term 2 LEADER]: Becoming Leader. State: Replica: 1cb67158133e4648a6eceba32d4dbba3, State: Running, Role: LEADER
I20240502 14:17:23.897055 14949 consensus_queue.cc:237] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
W20240502 14:17:23.901358 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:17:23.903844 14688 raft_consensus.cc:482] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:23.904019 14688 raft_consensus.cc:504] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:17:23.904750 14688 leader_election.cc:288] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:17:23.906503 26754 leader_election.cc:334] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:23.908707 26755 leader_election.cc:334] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:23.908843 26755 leader_election.cc:302] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:23.909111 14688 raft_consensus.cc:2726] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:17:23.909622 26798 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:17:23.912124 26278 catalog_manager.cc:5484] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 reported cstate change: term changed from 1 to 2, leader changed from 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) to 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197). New cstate: current_term: 2 leader_uuid: "1cb67158133e4648a6eceba32d4dbba3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } }
I20240502 14:17:23.942118 14726 raft_consensus.cc:482] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:23.942366 14726 raft_consensus.cc:504] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:17:23.943135 14726 leader_election.cc:288] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:17:23.948803 26625 leader_election.cc:334] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:23.949050 26624 leader_election.cc:334] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:23.949158 26624 leader_election.cc:302] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:23.949935 15099 raft_consensus.cc:2726] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:17:23.958920 14900 raft_consensus.cc:482] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:23.959111 14900 raft_consensus.cc:504] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:17:23.959781 14900 leader_election.cc:288] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:17:23.960881 29175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb065120d1304a2a81ce932934b911f3" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:17:23.961124 29175 raft_consensus.cc:2454] T eb065120d1304a2a81ce932934b911f3 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
I20240502 14:17:23.961805 26362 leader_election.cc:302] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:17:23.962491 14900 raft_consensus.cc:2781] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:17:23.962572 14900 raft_consensus.cc:482] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:23.962649 14900 raft_consensus.cc:3037] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:17:23.964540 26363 leader_election.cc:334] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:23.965217 14900 raft_consensus.cc:504] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:17:23.966413 29175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb065120d1304a2a81ce932934b911f3" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:17:23.966650 29175 raft_consensus.cc:3037] T eb065120d1304a2a81ce932934b911f3 P 1cb67158133e4648a6eceba32d4dbba3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:17:23.969091 29175 raft_consensus.cc:2454] T eb065120d1304a2a81ce932934b911f3 P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 2.
I20240502 14:17:23.965996 14900 leader_election.cc:288] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 election: Requested vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:17:23.969700 26362 leader_election.cc:302] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 89af7f423cf44717a08d6b142f0ed49d; no voters:
I20240502 14:17:23.970288 14900 raft_consensus.cc:2781] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:17:23.970434 14900 raft_consensus.cc:686] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d [term 2 LEADER]: Becoming Leader. State: Replica: 89af7f423cf44717a08d6b142f0ed49d, State: Running, Role: LEADER
W20240502 14:17:23.970980 26363 leader_election.cc:334] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:23.970980 14900 consensus_queue.cc:237] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:17:23.976807 26278 catalog_manager.cc:5484] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d reported cstate change: term changed from 1 to 2, leader changed from 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) to 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193). New cstate: current_term: 2 leader_uuid: "89af7f423cf44717a08d6b142f0ed49d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } }
W20240502 14:17:23.990885 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:23.993593 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:17:23.996925 14885 raft_consensus.cc:482] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:23.997102 14885 raft_consensus.cc:504] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:17:23.997757 14885 leader_election.cc:288] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:17:23.998260 29573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffaded0bc6204e8c8576ea8eabdd18ab" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86" is_pre_election: true
I20240502 14:17:23.998505 29573 raft_consensus.cc:2454] T ffaded0bc6204e8c8576ea8eabdd18ab P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
I20240502 14:17:23.999121 26755 leader_election.cc:302] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, e4cfb8fc5a474177af6848a6efa13a27; no voters:
W20240502 14:17:23.999382 26755 leader_election.cc:334] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:23.999408 14885 raft_consensus.cc:2781] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:17:23.999485 14885 raft_consensus.cc:482] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:23.999557 14885 raft_consensus.cc:3037] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:17:24.002599 14885 raft_consensus.cc:504] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:17:24.003296 14885 leader_election.cc:288] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 election: Requested vote from peers 4092902b3e674857a0147f1b4f2cdb86 (127.19.94.200:46097), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:17:24.003845 29574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffaded0bc6204e8c8576ea8eabdd18ab" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4092902b3e674857a0147f1b4f2cdb86"
I20240502 14:17:24.004170 29574 raft_consensus.cc:3037] T ffaded0bc6204e8c8576ea8eabdd18ab P 4092902b3e674857a0147f1b4f2cdb86 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:17:24.005805 26755 leader_election.cc:334] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:24.006876 29574 raft_consensus.cc:2454] T ffaded0bc6204e8c8576ea8eabdd18ab P 4092902b3e674857a0147f1b4f2cdb86 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 2.
I20240502 14:17:24.007364 26755 leader_election.cc:302] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4092902b3e674857a0147f1b4f2cdb86, e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab
I20240502 14:17:24.007673 14885 raft_consensus.cc:2781] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:17:24.007815 14885 raft_consensus.cc:686] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 [term 2 LEADER]: Becoming Leader. State: Replica: e4cfb8fc5a474177af6848a6efa13a27, State: Running, Role: LEADER
I20240502 14:17:24.008102 14885 consensus_queue.cc:237] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } }
I20240502 14:17:24.010015 15105 raft_consensus.cc:482] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:24.010222 15105 raft_consensus.cc:504] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:17:24.011004 15105 leader_election.cc:288] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:17:24.013839 26278 catalog_manager.cc:5484] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 reported cstate change: term changed from 1 to 2, leader changed from 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) to e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196). New cstate: current_term: 2 leader_uuid: "e4cfb8fc5a474177af6848a6efa13a27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } }
I20240502 14:17:24.015734 29175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f3cbedfa1da42698615a544d9dc3276" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
I20240502 14:17:24.016005 29175 raft_consensus.cc:2454] T 9f3cbedfa1da42698615a544d9dc3276 P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 2.
W20240502 14:17:24.017522 26494 leader_election.cc:334] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:24.018452 26493 leader_election.cc:302] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab
I20240502 14:17:24.018790 15105 raft_consensus.cc:2781] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 14:17:24.018895 15105 raft_consensus.cc:482] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:24.019001 15105 raft_consensus.cc:3037] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:17:24.020202 14726 raft_consensus.cc:482] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:24.020496 14726 raft_consensus.cc:504] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:17:24.021634 14726 leader_election.cc:288] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:17:24.022008 15105 raft_consensus.cc:504] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:17:24.022883 29175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce03ee354c54457487ea939301ccd90e" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3" is_pre_election: true
W20240502 14:17:24.023193 26625 leader_election.cc:334] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:24.023226 29175 raft_consensus.cc:2454] T ce03ee354c54457487ea939301ccd90e P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 2.
I20240502 14:17:24.024147 15105 leader_election.cc:288] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 3 election: Requested vote from peers 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:17:24.024490 29175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f3cbedfa1da42698615a544d9dc3276" candidate_uuid: "ba187e38a2754d60a12de9ac4856430a" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:17:24.024559 26624 leader_election.cc:302] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters: 68a3c2b17ed94e06aba66ed4178132ab
I20240502 14:17:24.024711 29175 raft_consensus.cc:3037] T 9f3cbedfa1da42698615a544d9dc3276 P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:17:24.025189 14726 raft_consensus.cc:2781] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 14:17:24.025323 14726 raft_consensus.cc:482] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:24.025471 14726 raft_consensus.cc:3037] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:17:24.027019 29175 raft_consensus.cc:2454] T 9f3cbedfa1da42698615a544d9dc3276 P 1cb67158133e4648a6eceba32d4dbba3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba187e38a2754d60a12de9ac4856430a in term 3.
I20240502 14:17:24.027976 26493 leader_election.cc:302] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, ba187e38a2754d60a12de9ac4856430a; no voters:
I20240502 14:17:24.028271 15105 raft_consensus.cc:2781] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:17:24.028525 15105 raft_consensus.cc:686] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a [term 3 LEADER]: Becoming Leader. State: Replica: ba187e38a2754d60a12de9ac4856430a, State: Running, Role: LEADER
W20240502 14:17:24.028573 26494 leader_election.cc:334] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:24.028941 15105 consensus_queue.cc:237] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:17:24.031587 14726 raft_consensus.cc:504] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
W20240502 14:17:24.036423 26625 leader_election.cc:334] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:24.037092 26278 catalog_manager.cc:5484] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a reported cstate change: term changed from 2 to 3, leader changed from 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) to ba187e38a2754d60a12de9ac4856430a (127.19.94.194). New cstate: current_term: 3 leader_uuid: "ba187e38a2754d60a12de9ac4856430a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } }
I20240502 14:17:24.039619 29175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce03ee354c54457487ea939301ccd90e" candidate_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "1cb67158133e4648a6eceba32d4dbba3"
I20240502 14:17:24.039978 29175 raft_consensus.cc:3037] T ce03ee354c54457487ea939301ccd90e P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:17:24.045903 29175 raft_consensus.cc:2454] T ce03ee354c54457487ea939301ccd90e P 1cb67158133e4648a6eceba32d4dbba3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ab78b2a1b74d2bbf2cdeafa0a4af23 in term 3.
I20240502 14:17:24.046531 26624 leader_election.cc:302] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cb67158133e4648a6eceba32d4dbba3, 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters: 68a3c2b17ed94e06aba66ed4178132ab
I20240502 14:17:24.046969 15099 raft_consensus.cc:2781] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:17:24.048310 14726 leader_election.cc:288] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 3 election: Requested vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 1cb67158133e4648a6eceba32d4dbba3 (127.19.94.197:38715)
I20240502 14:17:24.059514 15099 raft_consensus.cc:686] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 3 LEADER]: Becoming Leader. State: Replica: 69ab78b2a1b74d2bbf2cdeafa0a4af23, State: Running, Role: LEADER
W20240502 14:17:24.059674 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:17:24.059880 15099 consensus_queue.cc:237] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } }
I20240502 14:17:24.066932 26278 catalog_manager.cc:5484] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 reported cstate change: term changed from 2 to 3, leader changed from 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199) to 69ab78b2a1b74d2bbf2cdeafa0a4af23 (127.19.94.195). New cstate: current_term: 3 leader_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 } } }
W20240502 14:17:24.163337 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:24.163524 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:17:24.196609 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:24.246552 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:24.274902 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:24.329154 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:24.332624 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:17:24.338611 26558 raft_consensus.cc:1261] T 177a34c96aa84cb4aa50bbd79cab4144 P ba187e38a2754d60a12de9ac4856430a [term 2 FOLLOWER]: Refusing update from remote peer 1cb67158133e4648a6eceba32d4dbba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:17:24.339545 15100 consensus_queue.cc:1035] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:17:24.351197 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:24.358616 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:17:24.381176 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:17:24.390894 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:17:24.391359 29174 raft_consensus.cc:1261] T 01e6bc3689db46e2ad7d0484a29af287 P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:17:24.391973 15110 consensus_queue.cc:1035] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:17:24.470857 29573 raft_consensus.cc:1261] T ffaded0bc6204e8c8576ea8eabdd18ab P 4092902b3e674857a0147f1b4f2cdb86 [term 2 FOLLOWER]: Refusing update from remote peer e4cfb8fc5a474177af6848a6efa13a27: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:17:24.471547 14688 consensus_queue.cc:1035] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4092902b3e674857a0147f1b4f2cdb86" member_type: VOTER last_known_addr { host: "127.19.94.200" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:17:24.472678 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:17:24.476807 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:24.490468 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:17:24.492946 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:17:24.500085 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:17:24.542145 29174 raft_consensus.cc:1261] T 9f3cbedfa1da42698615a544d9dc3276 P 1cb67158133e4648a6eceba32d4dbba3 [term 3 FOLLOWER]: Refusing update from remote peer ba187e38a2754d60a12de9ac4856430a: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240502 14:17:24.542937 15105 consensus_queue.cc:1035] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 14:17:24.547941 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:17:24.582808 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:24.593899 29175 raft_consensus.cc:1261] T eb065120d1304a2a81ce932934b911f3 P 1cb67158133e4648a6eceba32d4dbba3 [term 2 FOLLOWER]: Refusing update from remote peer 89af7f423cf44717a08d6b142f0ed49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:17:24.595131 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:17:24.595170 15103 consensus_queue.cc:1035] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:17:24.658075 29175 raft_consensus.cc:1261] T ce03ee354c54457487ea939301ccd90e P 1cb67158133e4648a6eceba32d4dbba3 [term 3 FOLLOWER]: Refusing update from remote peer 69ab78b2a1b74d2bbf2cdeafa0a4af23: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240502 14:17:24.658818 15099 consensus_queue.cc:1035] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb67158133e4648a6eceba32d4dbba3" member_type: VOTER last_known_addr { host: "127.19.94.197" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:17:24.670746 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:24.695983 26362 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.197:38715, user_credentials={real_user=slave}} blocked reactor thread for 80421.1us
W20240502 14:17:24.698017 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:17:24.715526 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:17:24.725646 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:17:24.755136 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:24.778249 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:24.804690 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:24.846789 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:24.859889 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:24.862246 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:24.875406 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:17:24.885787 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:24.896257 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:17:25.003285 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:25.030401 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:25.054752 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:25.082284 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:17:25.088012 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:17:25.089340 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:17:25.089640 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:17:25.116293 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:17:25.132930 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:17:25.134199 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:17:25.150771 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:17:25.156685 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:17:25.158483 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:17:25.163645 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:25.165923 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:17:25.167374 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:17:25.176270 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:17:25.180675 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:17:25.184636 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:25.192438 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:17:25.205693 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:17:25.214407 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:17:25.229266 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:17:25.238559 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:25.243585 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:17:25.254295 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:25.263192 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:17:25.306247 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:17:25.313458 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:25.334221 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:17:25.336383 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:17:25.355966 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:25.399048 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:17:25.400904 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:25.406657 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:17:25.464762 15105 raft_consensus.cc:482] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:25.464994 15105 raft_consensus.cc:504] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:17:25.465790 15105 leader_election.cc:288] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
W20240502 14:17:25.465790 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:17:25.468271 26494 leader_election.cc:334] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:25.469736 26493 leader_election.cc:334] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:25.469884 26493 leader_election.cc:302] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:25.470532 15105 raft_consensus.cc:2726] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:17:25.490423 15205 raft_consensus.cc:482] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:25.490657 15205 raft_consensus.cc:504] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:17:25.491529 15205 leader_election.cc:288] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:17:25.493227 26754 leader_election.cc:334] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:25.494663 26755 leader_election.cc:334] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:25.494815 26755 leader_election.cc:302] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:25.495287 15205 raft_consensus.cc:2726] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:17:25.500674 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:17:25.540055 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:25.544080 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:17:25.594456 14688 raft_consensus.cc:482] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:25.594696 14688 raft_consensus.cc:504] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
W20240502 14:17:25.595278 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:17:25.595623 14688 leader_election.cc:288] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:17:25.597280 26754 leader_election.cc:334] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:25.598052 26755 leader_election.cc:334] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:25.598194 26755 leader_election.cc:302] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:25.598605 15205 raft_consensus.cc:2726] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:17:25.635921 15108 raft_consensus.cc:482] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:25.636153 15108 raft_consensus.cc:504] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:17:25.636960 15108 leader_election.cc:288] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:17:25.640162 26493 leader_election.cc:334] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:25.641072 26494 leader_election.cc:334] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:25.641266 26494 leader_election.cc:302] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:25.641975 15108 raft_consensus.cc:2726] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:17:25.734544 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:25.737535 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:25.760335 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:17:25.768761 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:17:25.790553 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:17:25.799604 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:25.815562 15103 raft_consensus.cc:482] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:25.815853 15103 raft_consensus.cc:504] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:17:25.816963 15103 leader_election.cc:288] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:17:25.822280 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:17:25.823781 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:25.824767 26362 leader_election.cc:334] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:25.826531 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:17:25.827698 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:25.847299 26363 leader_election.cc:334] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:25.847458 26363 leader_election.cc:302] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:25.847713 15103 raft_consensus.cc:2726] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:17:25.882685 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:25.896871 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:17:25.898380 15103 raft_consensus.cc:482] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:25.898538 15103 raft_consensus.cc:504] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:17:25.899312 15103 leader_election.cc:288] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:17:25.900179 26819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cea6495041642f58e1eb5d26fc10df5" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:17:25.900480 26819 raft_consensus.cc:2454] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
W20240502 14:17:25.903995 26363 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:25.907686 26362 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:25.907835 26362 leader_election.cc:302] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:25.908160 15103 raft_consensus.cc:2726] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:17:25.997687 15108 raft_consensus.cc:482] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:25.997890 15108 raft_consensus.cc:504] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:17:25.998674 15108 leader_election.cc:288] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:17:26.001351 26494 leader_election.cc:334] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:26.001540 26493 leader_election.cc:334] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:26.001683 26493 leader_election.cc:302] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:26.001986 15350 raft_consensus.cc:2726] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:17:26.031546 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:26.038096 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:26.122166 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:17:26.152884 15198 raft_consensus.cc:482] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:26.153100 15198 raft_consensus.cc:504] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:17:26.153853 15198 leader_election.cc:288] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:17:26.161890 26624 leader_election.cc:334] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:26.175987 26625 leader_election.cc:334] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:26.176122 26625 leader_election.cc:302] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:26.176710 15198 raft_consensus.cc:2726] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:17:26.211230 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:17:26.234966 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:17:26.248309 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:26.252496 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:17:26.261335 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:26.269762 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:26.296993 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:17:26.301023 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:17:26.301836 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:26.306219 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:17:26.336683 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:26.354809 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:26.369319 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:26.466516 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:26.507809 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:26.550117 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:26.581336 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:26.594904 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:26.680454 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:26.707269 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:26.757702 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:26.833205 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:17:26.834365 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:17:26.834671 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:17:26.865430 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:26.868574 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:26.869211 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:17:26.871387 29222 maintenance_manager.cc:419] P 1cb67158133e4648a6eceba32d4dbba3: Scheduling FlushMRSOp(39a61744642f40c8ae92da708dc91ca3): perf score=0.033391
I20240502 14:17:26.878559 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:26.892480 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:17:26.919795 26493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 171 similar messages]
W20240502 14:17:26.921316 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:26.927104 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:17:26.946127 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:17:26.949111 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:17:26.971057 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:26.972265 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:26.977967 26754 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 138 similar messages]
I20240502 14:17:26.997776 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:17:26.999522 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:27.054379 26363 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 94 similar messages]
W20240502 14:17:27.069536 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:17:27.097875 29113 maintenance_manager.cc:643] P 1cb67158133e4648a6eceba32d4dbba3: FlushMRSOp(39a61744642f40c8ae92da708dc91ca3) complete. Timing: real 0.226s user 0.045s sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":22163,"dirs.run_cpu_time_us":199,"dirs.run_wall_time_us":2160,"drs_written":1,"lbm_read_time_us":137,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3504,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":26584,"threads_started":2,"wal-append.queue_time_us":2172}
I20240502 14:17:27.100802 29222 maintenance_manager.cc:419] P 1cb67158133e4648a6eceba32d4dbba3: Scheduling UndoDeltaBlockGCOp(39a61744642f40c8ae92da708dc91ca3): 4586 bytes on disk
I20240502 14:17:27.101511 29113 maintenance_manager.cc:643] P 1cb67158133e4648a6eceba32d4dbba3: UndoDeltaBlockGCOp(39a61744642f40c8ae92da708dc91ca3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
W20240502 14:17:27.115254 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:17:27.124904 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:17:27.147681 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:27.156293 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:27.173955 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:27.286289 29111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240502 14:17:27.305204 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:27.312888 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:27.334424 26625 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240502 14:17:27.394738 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:27.395167 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:27.396148 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:27.398710 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:17:27.398963 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:17:27.427949 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:27.438920 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:27.442309 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:17:27.475809 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:27.482270 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:27.483217 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:17:27.484724 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:17:27.493240 15386 raft_consensus.cc:482] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:27.493407 15386 raft_consensus.cc:504] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:17:27.494151 15386 leader_election.cc:288] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:17:27.495424 26819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cea6495041642f58e1eb5d26fc10df5" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:17:27.495657 26819 raft_consensus.cc:2454] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
W20240502 14:17:27.497956 26363 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:27.512071 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:27.512979 26362 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:27.513137 26362 leader_election.cc:302] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:27.517668 15103 raft_consensus.cc:2726] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:17:27.519460 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:27.533181 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:27.541914 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:27.576488 29511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240502 14:17:27.579609 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:27.592039 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:27.595122 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:27.614156 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:17:27.623813 15105 raft_consensus.cc:482] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:27.623998 15105 raft_consensus.cc:504] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:17:27.624727 15105 leader_election.cc:288] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
W20240502 14:17:27.626710 26493 leader_election.cc:334] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:27.628782 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:27.630755 26494 leader_election.cc:334] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:27.630905 26494 leader_election.cc:302] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:27.631500 15105 raft_consensus.cc:2726] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:17:27.654318 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:27.712374 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:27.760079 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:27.768360 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:27.771314 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:27.782649 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:27.795929 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:27.842355 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:27.848125 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:27.861105 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:27.863474 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:17:27.950441 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:27.965104 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:27.968250 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:27.979404 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:17:27.980661 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:17:27.981932 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:17:27.987346 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:17:27.992419 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:17:28.018381 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:28.030843 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:28.037982 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:17:28.042312 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:28.076184 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:28.079772 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:28.099114 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:28.107080 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:17:28.113955 29222 maintenance_manager.cc:419] P 1cb67158133e4648a6eceba32d4dbba3: Scheduling FlushMRSOp(9f3cbedfa1da42698615a544d9dc3276): perf score=0.033380
W20240502 14:17:28.129701 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:28.149847 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:17:28.161549 29113 maintenance_manager.cc:643] P 1cb67158133e4648a6eceba32d4dbba3: FlushMRSOp(9f3cbedfa1da42698615a544d9dc3276) complete. Timing: real 0.047s user 0.031s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":426,"dirs.run_cpu_time_us":287,"dirs.run_wall_time_us":1788,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":816,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":4191,"threads_started":2,"wal-append.queue_time_us":1890}
I20240502 14:17:28.164124 29222 maintenance_manager.cc:419] P 1cb67158133e4648a6eceba32d4dbba3: Scheduling UndoDeltaBlockGCOp(9f3cbedfa1da42698615a544d9dc3276): 4925 bytes on disk
I20240502 14:17:28.164844 29113 maintenance_manager.cc:643] P 1cb67158133e4648a6eceba32d4dbba3: UndoDeltaBlockGCOp(9f3cbedfa1da42698615a544d9dc3276) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":119,"lbm_reads_lt_1ms":4}
W20240502 14:17:28.204228 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:17:28.273741 14688 raft_consensus.cc:482] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:28.274003 14688 raft_consensus.cc:504] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:17:28.274797 14688 leader_election.cc:288] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:17:28.276757 26755 leader_election.cc:334] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:28.277470 26754 leader_election.cc:334] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:28.277609 26754 leader_election.cc:302] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:28.277897 14688 raft_consensus.cc:2726] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:17:28.376628 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:28.414229 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:28.476655 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:17:28.497813 15636 raft_consensus.cc:482] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:28.498008 15636 raft_consensus.cc:504] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
W20240502 14:17:28.502736 26363 leader_election.cc:334] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:28.503131 26362 leader_election.cc:334] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:28.503248 26362 leader_election.cc:302] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:28.503782 15386 raft_consensus.cc:2726] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:17:28.505793 15636 leader_election.cc:288] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:17:28.538756 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:17:28.538866 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:17:28.548473 15446 raft_consensus.cc:482] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:28.548692 15446 raft_consensus.cc:504] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:17:28.549408 15446 leader_election.cc:288] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:17:28.551487 26755 leader_election.cc:334] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:28.552222 26754 leader_election.cc:334] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:28.552346 26754 leader_election.cc:302] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:28.552788 15446 raft_consensus.cc:2726] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:17:28.564450 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:17:28.566761 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:17:28.580700 15105 raft_consensus.cc:482] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:28.580905 15105 raft_consensus.cc:504] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:17:28.581647 15105 leader_election.cc:288] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:17:28.587143 26493 leader_election.cc:334] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:28.588799 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:28.589099 26494 leader_election.cc:334] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:28.589226 26494 leader_election.cc:302] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:28.591734 15108 raft_consensus.cc:2726] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:17:28.596904 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:28.606412 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:28.659556 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:17:28.667951 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:17:28.680338 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:28.702699 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:28.757419 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:28.768688 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:17:28.792380 15105 raft_consensus.cc:482] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:28.792595 15105 raft_consensus.cc:504] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:17:28.793206 15105 leader_election.cc:288] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:17:28.794610 26494 leader_election.cc:334] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:28.794829 26493 leader_election.cc:334] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:28.794951 26493 leader_election.cc:302] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:28.795174 15105 raft_consensus.cc:2726] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:17:28.806618 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:28.834442 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:28.882283 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:28.934581 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:28.936192 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:28.969540 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:28.974356 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:28.990965 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:17:29.053552 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:29.054076 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:29.064126 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:29.075014 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:17:29.078794 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:17:29.095191 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:17:29.112596 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:29.148634 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:17:29.178897 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:17:29.190403 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:29.220587 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:29.241415 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:29.270360 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:29.296936 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:29.309825 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:29.333545 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:17:29.338313 15708 raft_consensus.cc:482] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:29.338515 15708 raft_consensus.cc:504] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:17:29.339177 15708 leader_election.cc:288] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:17:29.340817 26625 leader_election.cc:334] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:29.340960 26624 leader_election.cc:334] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:29.341065 26624 leader_election.cc:302] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:29.341351 15708 raft_consensus.cc:2726] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:17:29.419549 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:29.434685 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:29.456040 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:29.462255 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:29.470209 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:29.549345 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:29.551337 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:29.576131 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:29.600781 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:29.614199 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:29.630662 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:29.676420 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:17:29.697767 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:17:29.700371 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:17:29.709141 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:17:29.714601 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:17:29.721906 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:29.731179 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:29.744205 15108 raft_consensus.cc:482] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:29.744469 15108 raft_consensus.cc:504] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:17:29.745252 15108 leader_election.cc:288] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
W20240502 14:17:29.747351 26494 leader_election.cc:334] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:29.779404 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:29.784085 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:29.812191 26493 leader_election.cc:334] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:29.812510 26493 leader_election.cc:302] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:29.816270 15108 raft_consensus.cc:2726] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:17:29.817363 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:29.821883 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:29.822777 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:29.864779 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:29.873198 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:29.901852 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:29.913913 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:29.928476 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:29.982859 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:17:29.985746 15644 raft_consensus.cc:482] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
W20240502 14:17:29.985965 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:17:29.985930 15644 raft_consensus.cc:504] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:17:29.986835 15644 leader_election.cc:288] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:17:29.988312 26819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cea6495041642f58e1eb5d26fc10df5" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:17:29.988582 26819 raft_consensus.cc:2454] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
W20240502 14:17:29.990626 26363 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:29.994554 26362 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:29.994663 26362 leader_election.cc:302] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:29.995209 15103 raft_consensus.cc:2726] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:17:30.074090 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:30.144606 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:30.150913 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:17:30.164260 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:17:30.164672 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:17:30.164867 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:30.178148 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:30.184016 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:30.187842 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:17:30.193548 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:30.207695 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:17:30.212425 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:30.234097 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:30.238472 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:30.254436 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:30.291373 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:30.296010 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:30.321597 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:30.326153 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:30.337644 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:30.344836 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:30.354640 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:30.402014 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:30.408093 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:30.421588 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:30.427822 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:30.478011 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:30.536607 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:30.582702 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:30.701042 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:30.708921 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:17:30.728921 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:17:30.747360 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:30.747396 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:17:30.761566 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:17:30.769171 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:17:30.782841 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:17:30.784669 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:30.849217 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:30.861927 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:17:30.878329 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:30.951885 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:30.953931 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:30.979715 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:31.071880 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:31.122496 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:31.155265 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:31.155416 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:31.163426 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:31.169600 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:31.194828 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:31.218143 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:17:31.229234 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:17:31.278381 15103 raft_consensus.cc:482] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:31.278563 15103 raft_consensus.cc:504] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:17:31.279204 15103 leader_election.cc:288] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:17:31.283856 26363 leader_election.cc:334] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:31.285910 26362 leader_election.cc:334] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:31.286078 26362 leader_election.cc:302] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:31.286725 15103 raft_consensus.cc:2726] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:17:31.291702 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:31.300933 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:31.325026 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:17:31.329993 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:31.330929 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:17:31.334542 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:31.344197 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:17:31.348009 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:31.348052 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:17:31.361001 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:17:31.393509 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:31.412624 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:31.514117 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:17:31.540028 15103 raft_consensus.cc:482] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:31.540253 15103 raft_consensus.cc:504] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:17:31.541361 15103 leader_election.cc:288] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:17:31.542490 26819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cea6495041642f58e1eb5d26fc10df5" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:17:31.542737 26819 raft_consensus.cc:2454] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
W20240502 14:17:31.542986 26363 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:31.554613 26362 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:31.554771 26362 leader_election.cc:302] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:31.555157 15880 raft_consensus.cc:2726] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:17:31.589526 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:31.598989 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:31.609315 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:31.611354 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:31.614174 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:31.672796 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:31.747483 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:31.778947 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:31.830019 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:31.843416 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:17:31.856861 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:17:31.857761 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:31.905185 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:17:31.906033 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:17:31.910154 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:17:31.925133 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:31.937405 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:31.954008 26494 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 178 similar messages]
W20240502 14:17:32.005048 26755 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 144 similar messages]
W20240502 14:17:32.032703 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:32.034623 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:17:32.059053 15446 raft_consensus.cc:482] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:32.059247 15446 raft_consensus.cc:504] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:17:32.059959 15446 leader_election.cc:288] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:17:32.063149 26754 leader_election.cc:334] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:32.068848 26755 leader_election.cc:334] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:32.068993 26755 leader_election.cc:302] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:32.069257 15446 raft_consensus.cc:2726] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:17:32.079083 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:32.091858 26362 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 97 similar messages]
W20240502 14:17:32.104914 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:32.125226 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:17:32.137202 15954 raft_consensus.cc:482] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:32.137406 15954 raft_consensus.cc:504] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:17:32.138098 15954 leader_election.cc:288] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:17:32.140159 26493 leader_election.cc:334] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:32.140448 26494 leader_election.cc:334] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:32.140560 26494 leader_election.cc:302] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:32.141032 15108 raft_consensus.cc:2726] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:17:32.143311 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:32.164570 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:32.167413 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:17:32.198875 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:32.231590 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:32.277455 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:32.286031 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:32.286435 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:32.287153 29111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240502 14:17:32.290242 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:32.332499 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:32.362089 26624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 103 similar messages]
W20240502 14:17:32.363087 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:32.372267 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:17:32.410480 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:17:32.411808 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:17:32.425253 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:17:32.464475 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:17:32.476231 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:32.481345 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:17:32.499991 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:32.524422 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:32.525890 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:32.528245 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:32.542526 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:17:32.545668 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:32.552644 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:32.563604 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:17:32.575762 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:32.589941 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:32.645159 29510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 137 similar messages]
W20240502 14:17:32.662955 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:32.706727 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:32.710798 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:32.741154 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:32.796774 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:32.800429 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:32.841719 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:32.887406 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:17:32.892077 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:32.924892 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:32.939839 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:32.955457 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:32.991207 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:32.993530 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:17:32.998781 15108 raft_consensus.cc:482] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:32.999018 15108 raft_consensus.cc:504] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:17:32.999954 15108 leader_election.cc:288] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:17:33.002029 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:33.003957 26494 leader_election.cc:334] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:33.004348 26493 leader_election.cc:334] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:33.004487 26493 leader_election.cc:302] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:33.005117 15954 raft_consensus.cc:2726] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:17:33.027415 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:17:33.042775 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:17:33.043272 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:17:33.054669 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:33.064519 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:33.081089 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:17:33.084862 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:17:33.090914 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:33.153752 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:33.154212 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:33.161015 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:17:33.213497 16073 raft_consensus.cc:482] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:33.213727 16073 raft_consensus.cc:504] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:17:33.214278 16073 leader_election.cc:288] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:17:33.216358 26755 leader_election.cc:334] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:33.218623 26754 leader_election.cc:334] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:33.218762 26754 leader_election.cc:302] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:33.219002 16073 raft_consensus.cc:2726] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:17:33.236887 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:33.251976 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:17:33.266207 15735 raft_consensus.cc:482] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:33.266399 15735 raft_consensus.cc:504] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:17:33.267056 15735 leader_election.cc:288] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:17:33.269075 26624 leader_election.cc:334] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:33.269235 26625 leader_election.cc:334] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:33.269343 26625 leader_election.cc:302] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:33.269572 15735 raft_consensus.cc:2726] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:17:33.318648 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:33.346792 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:33.354576 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:33.379614 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:33.430258 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:33.442680 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:33.557140 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:17:33.573736 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:33.584798 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:33.604058 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:17:33.604497 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:17:33.607650 15108 raft_consensus.cc:482] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:33.607847 15108 raft_consensus.cc:504] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:17:33.608577 15108 leader_election.cc:288] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
W20240502 14:17:33.610685 26493 leader_election.cc:334] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:33.625516 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:33.644662 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:17:33.648625 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:17:33.648680 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:17:33.656900 26494 leader_election.cc:334] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:33.657162 26494 leader_election.cc:302] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
W20240502 14:17:33.657248 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:17:33.657681 15954 raft_consensus.cc:2726] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:17:33.661592 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:33.666347 15446 raft_consensus.cc:482] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:33.666546 15446 raft_consensus.cc:504] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:17:33.667356 15446 leader_election.cc:288] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:17:33.669728 26755 leader_election.cc:334] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:33.678048 26754 leader_election.cc:334] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:33.678226 26754 leader_election.cc:302] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:33.678848 15446 raft_consensus.cc:2726] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:17:33.684834 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:33.714267 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:17:33.727497 16073 raft_consensus.cc:482] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:33.727725 16073 raft_consensus.cc:504] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:17:33.728667 16073 leader_election.cc:288] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:17:33.729831 26427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cea6495041642f58e1eb5d26fc10df5" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:17:33.730116 26427 raft_consensus.cc:2454] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
W20240502 14:17:33.731542 26755 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:33.732734 26754 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:33.732869 26754 leader_election.cc:302] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:33.733356 16073 raft_consensus.cc:2726] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:17:33.814195 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:17:33.818560 16158 raft_consensus.cc:482] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:33.818771 16158 raft_consensus.cc:504] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:17:33.819499 16158 leader_election.cc:288] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:17:33.822744 26363 leader_election.cc:334] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:33.823429 26362 leader_election.cc:334] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:33.823560 26362 leader_election.cc:302] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:33.823946 16158 raft_consensus.cc:2726] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:17:33.856745 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:17:33.891695 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:17:33.893594 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:17:33.901091 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:17:33.902829 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:17:33.937907 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:17:33.949193 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:17:33.994307 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:34.061025 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:34.078141 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:34.095172 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:34.112792 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:17:34.120697 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:34.159430 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:34.239086 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:34.295056 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:34.296128 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:34.349751 26493 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.198:43659, user_credentials={real_user=slave}} blocked reactor thread for 54688.6us
W20240502 14:17:34.392627 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:34.422716 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:17:34.490275 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:34.535346 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:17:34.539485 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:34.544447 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:17:34.605036 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:17:34.630808 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:34.633257 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:17:34.641470 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:17:34.643090 26363 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.200:46097, user_credentials={real_user=slave}} blocked reactor thread for 48072.2us
I20240502 14:17:34.644824 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:34.649379 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:17:34.655308 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:17:34.672997 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:17:34.697692 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:17:34.705467 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:17:34.711863 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:34.741503 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:34.754194 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:17:34.755513 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:34.780210 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:17:34.837961 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:17:34.865049 15954 raft_consensus.cc:482] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:34.865259 15954 raft_consensus.cc:504] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:17:34.866019 15954 leader_election.cc:288] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:17:34.867897 26494 leader_election.cc:334] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:34.869043 26493 leader_election.cc:334] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:34.869167 26493 leader_election.cc:302] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:34.869455 15954 raft_consensus.cc:2726] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:17:34.872751 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:34.876657 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:34.918159 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:17:34.999269 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:17:35.008337 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:35.014344 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:35.051330 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:35.059799 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:35.086970 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:35.099622 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:35.112195 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:17:35.121412 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:17:35.123966 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:17:35.134724 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:17:35.143306 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:17:35.152722 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:17:35.154119 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:35.155093 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:17:35.175694 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:17:35.180922 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:17:35.189158 16073 raft_consensus.cc:482] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:35.189347 16073 raft_consensus.cc:504] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:17:35.190023 16073 leader_election.cc:288] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:17:35.194185 26754 leader_election.cc:334] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:35.194561 26755 leader_election.cc:334] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:35.194679 26755 leader_election.cc:302] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:35.195065 16073 raft_consensus.cc:2726] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:17:35.202343 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:35.269258 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:35.287667 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:17:35.288321 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:35.299503 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:35.365367 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:17:35.369573 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:17:35.392446 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:35.447814 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:35.454178 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:35.461473 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:35.471585 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:35.514757 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:17:35.527665 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:35.533695 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:35.535374 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:35.536266 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:35.652138 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:35.671707 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:17:35.685460 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:17:35.741782 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:17:35.748628 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:17:35.765491 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:17:35.769644 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:17:35.780627 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:35.812578 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:35.843681 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:17:35.853020 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:35.853399 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:17:35.854974 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:35.868351 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:35.900646 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:17:35.905490 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:17:35.918850 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:35.946981 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:36.059432 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:17:36.208722 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:17:36.255208 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:17:36.274031 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:36.292846 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:17:36.298713 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:17:36.310470 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:17:36.398550 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:17:36.402657 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:17:36.403470 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:36.421519 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:17:36.433554 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:36.446041 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:17:36.457219 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:36.476073 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:17:36.486536 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:17:36.501400 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:36.519554 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:36.543622 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:17:36.547188 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:36.594424 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:17:36.610209 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:36.613509 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:17:36.632994 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:17:36.678947 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:17:36.752256 15387 raft_consensus.cc:482] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:36.752475 15387 raft_consensus.cc:504] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:17:36.753198 15387 leader_election.cc:288] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:17:36.756479 26625 leader_election.cc:334] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:36.757581 26624 leader_election.cc:334] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:36.757707 26624 leader_election.cc:302] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:36.758189 15387 raft_consensus.cc:2726] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:17:36.770764 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:36.800539 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:17:36.863487 16073 raft_consensus.cc:482] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:36.863745 16073 raft_consensus.cc:504] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:17:36.864889 16073 leader_election.cc:288] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
W20240502 14:17:36.867545 26755 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:36.870602 26427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cea6495041642f58e1eb5d26fc10df5" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:17:36.870847 26427 raft_consensus.cc:2454] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
W20240502 14:17:36.873024 26754 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:36.873178 26754 leader_election.cc:302] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:36.873463 16073 raft_consensus.cc:2726] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:17:36.881532 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:17:36.923143 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:17:36.932529 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:17:36.939724 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:36.952070 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:17:36.964359 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:17:36.973050 26493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 181 similar messages]
W20240502 14:17:36.983150 26755 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 57337.2us
I20240502 14:17:36.993903 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:37.002302 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:17:37.010301 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:37.015810 26754 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 155 similar messages]
I20240502 14:17:37.018869 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:37.021442 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:17:37.028210 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:17:37.076254 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:37.082247 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:17:37.083844 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:37.085099 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:37.153261 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:17:37.155752 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:17:37.165117 26363 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 96 similar messages]
W20240502 14:17:37.166457 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:37.235626 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:37.313215 29112 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 14:17:37.334122 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:17:37.413700 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:17:37.426353 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:17:37.469991 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:17:37.472820 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:37.498000 26625 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 104 similar messages]
W20240502 14:17:37.499712 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:17:37.524230 16500 raft_consensus.cc:482] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:37.525040 16500 raft_consensus.cc:504] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:17:37.526015 16500 leader_election.cc:288] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:17:37.530176 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:17:37.542392 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:37.573031 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:37.591552 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:17:37.603228 26754 leader_election.cc:334] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:37.613468 26755 leader_election.cc:334] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:37.613799 26755 leader_election.cc:302] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:37.615634 15446 raft_consensus.cc:2726] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:17:37.671685 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:37.682235 29510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 135 similar messages]
W20240502 14:17:37.707052 26494 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 237360us
W20240502 14:17:37.713305 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:37.714423 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:17:37.714566 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:17:37.715050 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:37.723335 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:17:37.728639 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:17:37.728701 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:17:37.729480 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:17:37.741948 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:17:37.752426 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:17:37.766474 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:37.783205 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:17:37.784435 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:37.788467 26493 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.197:38715, user_credentials={real_user=slave}} blocked reactor thread for 70799.7us
W20240502 14:17:37.788925 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:37.823953 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:37.832895 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:17:37.836663 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:37.851459 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:37.883697 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:37.907289 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:37.930150 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:37.952775 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:38.031181 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:38.099334 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:38.117002 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:17:38.133376 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:17:38.138983 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:17:38.146153 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:17:38.162065 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:17:38.173578 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:38.182065 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:17:38.189993 16570 raft_consensus.cc:482] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:38.190227 16570 raft_consensus.cc:504] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:17:38.191241 16570 leader_election.cc:288] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:17:38.193321 26494 leader_election.cc:334] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:38.194483 26493 leader_election.cc:334] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:38.194614 26493 leader_election.cc:302] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:38.195261 16571 raft_consensus.cc:2726] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:17:38.202108 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:17:38.253998 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:17:38.283355 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:38.284647 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:38.285864 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:17:38.287509 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:38.307629 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:17:38.307768 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:17:38.328732 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:38.401067 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:17:38.429620 16563 raft_consensus.cc:482] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:38.429841 16563 raft_consensus.cc:504] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:17:38.430644 16563 leader_election.cc:288] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:17:38.433131 26362 leader_election.cc:334] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:38.449095 26363 leader_election.cc:334] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:38.449381 26363 leader_election.cc:302] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:38.449717 16563 raft_consensus.cc:2726] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:17:38.496053 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:17:38.538077 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:17:38.558925 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:38.618764 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:17:38.627496 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:38.639348 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:38.646061 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:17:38.798274 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:17:38.834671 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:17:38.849661 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:38.851258 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:17:38.910755 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:17:38.966864 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:38.971786 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:17:38.975338 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:38.989923 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:17:38.999258 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:39.024375 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:39.027770 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:17:39.042927 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:17:39.061442 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:17:39.078824 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:17:39.114288 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:39.153849 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:39.167773 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:17:39.189440 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:17:39.232461 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:39.274879 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:39.280591 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:17:39.283365 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:17:39.327909 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:39.335413 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:17:39.375969 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:39.381412 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:17:39.468164 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:17:39.476926 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:39.488929 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:17:39.535544 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:17:39.535728 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:17:39.535854 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:17:39.536684 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:39.544476 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:17:39.561539 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:39.590375 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:17:39.609483 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:39.610544 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:17:39.643235 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:39.644942 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:39.699272 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:17:39.755597 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:17:39.830476 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:39.839035 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:17:39.848737 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:39.952682 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:39.984340 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:39.987709 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:17:39.998494 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:40.005524 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:17:40.007741 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:17:40.014010 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:40.023756 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:17:40.040596 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:17:40.041643 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:17:40.046633 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:40.055326 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:17:40.064991 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:40.097968 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:40.099406 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:40.169219 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:40.251163 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:17:40.260887 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:17:40.266616 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:17:40.289533 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:40.296047 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:17:40.312287 16266 raft_consensus.cc:482] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:40.312508 16266 raft_consensus.cc:504] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:17:40.313536 16266 leader_election.cc:288] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:17:40.314016 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:17:40.315423 26363 leader_election.cc:334] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:40.316011 26362 leader_election.cc:334] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:40.316171 26362 leader_election.cc:302] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:40.316450 16266 raft_consensus.cc:2726] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:17:40.346884 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:17:40.350499 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:40.353488 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:17:40.358696 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:40.366307 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:40.378980 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:17:40.380276 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:17:40.445263 16501 raft_consensus.cc:482] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:40.445457 16501 raft_consensus.cc:504] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:17:40.446458 16501 leader_election.cc:288] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:17:40.449237 26427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cea6495041642f58e1eb5d26fc10df5" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
W20240502 14:17:40.449421 26754 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:40.449615 26427 raft_consensus.cc:2454] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
W20240502 14:17:40.449822 26755 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:40.449915 26755 leader_election.cc:302] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:40.450218 16501 raft_consensus.cc:2726] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:17:40.460991 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:40.495828 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:40.496167 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:40.517896 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:17:40.611901 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:40.616525 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:40.656745 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:40.666121 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:40.691023 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:40.696125 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:17:40.770778 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:17:40.780354 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:40.782064 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:17:40.829872 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:17:40.833690 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:17:40.873324 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:17:40.880843 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:40.881291 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:40.902806 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:17:40.903050 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:17:40.913508 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:40.959360 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:17:41.050141 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:17:41.117987 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:41.119774 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:17:41.148821 16570 raft_consensus.cc:482] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:41.149008 16570 raft_consensus.cc:504] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:17:41.149699 16570 leader_election.cc:288] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:17:41.158783 26494 leader_election.cc:334] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:41.168956 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:17:41.170356 26493 leader_election.cc:334] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:41.170500 26493 leader_election.cc:302] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:41.170897 16570 raft_consensus.cc:2726] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:17:41.181079 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:41.244364 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:17:41.257508 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:17:41.265017 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:41.296964 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:17:41.374462 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:17:41.374598 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:17:41.397907 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:17:41.420111 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:41.426648 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:17:41.432866 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:41.434345 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:17:41.454843 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:17:41.477011 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:17:41.500036 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:17:41.505921 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:17:41.526981 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:17:41.537387 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:17:41.556104 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:41.584738 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:17:41.643891 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:17:41.688241 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:41.743988 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:41.747496 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:41.759356 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:41.761725 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:41.813302 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:17:41.820338 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:17:41.826540 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:17:41.849717 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:17:41.863600 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:17:41.883929 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:41.941980 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:17:41.982543 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:41.991027 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:17:41.992368 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:41.994032 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:17:41.999166 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:42.009306 26493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 171 similar messages]
I20240502 14:17:42.024317 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:17:42.030370 26755 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240502 14:17:42.065255 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:17:42.090770 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:42.168551 26363 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 14:17:42.176380 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:17:42.188838 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:42.227595 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:42.319576 29111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240502 14:17:42.358911 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:17:42.363973 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:42.391047 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:17:42.406615 16570 raft_consensus.cc:482] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:42.406817 16570 raft_consensus.cc:504] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:17:42.407531 16570 leader_election.cc:288] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
W20240502 14:17:42.412271 26494 leader_election.cc:334] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:42.414129 26493 leader_election.cc:334] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:42.414261 26493 leader_election.cc:302] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:42.414777 16558 raft_consensus.cc:2726] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:17:42.416741 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:42.424263 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:17:42.481745 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:17:42.482177 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:17:42.493384 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:17:42.496441 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:17:42.526258 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:42.528625 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:17:42.530073 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:17:42.546212 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:42.558120 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:17:42.560550 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:17:42.572587 26624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 103 similar messages]
W20240502 14:17:42.586807 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:17:42.586829 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:17:42.613507 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:42.618831 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:42.623145 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:17:42.624882 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:42.633502 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:42.669567 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:42.671895 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:42.674968 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:42.719386 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:42.743242 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:42.754103 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:17:42.762568 29510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 147 similar messages]
W20240502 14:17:42.770362 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:42.800951 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:17:42.843322 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:17:42.845064 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:42.845619 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:42.873509 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:17:42.914338 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:17:42.915946 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:42.925460 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:17:43.043349 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:43.044750 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:17:43.045845 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:17:43.057300 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:17:43.066517 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:17:43.068667 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:43.081298 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:17:43.097857 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:43.131139 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:43.134755 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:43.217278 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:43.245885 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:17:43.279639 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:43.315351 16983 raft_consensus.cc:482] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:43.315605 16983 raft_consensus.cc:504] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:17:43.316326 16983 leader_election.cc:288] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:17:43.319896 26755 leader_election.cc:334] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:43.320755 26754 leader_election.cc:334] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:43.320864 26754 leader_election.cc:302] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:43.321398 16983 raft_consensus.cc:2726] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:17:43.325151 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:43.347661 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:43.379624 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:17:43.380446 16983 raft_consensus.cc:482] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:43.380615 16983 raft_consensus.cc:504] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:17:43.381417 16983 leader_election.cc:288] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:17:43.382449 26424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cea6495041642f58e1eb5d26fc10df5" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:17:43.382674 26424 raft_consensus.cc:2454] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
W20240502 14:17:43.390741 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:43.392441 26755 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:43.393991 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:17:43.410826 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:17:43.415138 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:43.422971 26754 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:43.423117 26754 leader_election.cc:302] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:43.423346 16983 raft_consensus.cc:2726] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:17:43.428431 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:17:43.501935 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:17:43.538815 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:17:43.561322 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:17:43.599707 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:43.607676 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:43.623991 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:17:43.625834 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:43.651854 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:17:43.656003 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:17:43.667538 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:43.708580 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:17:43.785877 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:17:43.827894 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:17:43.852308 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:43.879487 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:17:43.883330 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:17:43.926470 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:17:44.010579 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:44.051751 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:17:44.052726 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:17:44.054337 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:17:44.055393 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:44.057312 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:17:44.069237 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:17:44.089107 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:17:44.090209 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:17:44.094494 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:17:44.098774 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:17:44.108886 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:44.132093 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:17:44.175305 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:17:44.232417 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:44.302045 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:17:44.318872 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:44.380867 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:17:44.416599 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:17:44.466853 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:44.473244 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:17:44.512964 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:44.524031 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:44.555083 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:44.576318 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:44.628432 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:17:44.650957 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:44.663013 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:17:44.669492 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:17:44.684515 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:17:44.689882 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:44.713064 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:17:44.718017 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:17:44.725024 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:17:44.730859 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:44.769397 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:17:44.777536 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:44.811908 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:17:44.837427 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:17:44.889385 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:44.890404 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:44.912987 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:17:44.992138 17193 raft_consensus.cc:482] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:44.992337 17193 raft_consensus.cc:504] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:17:44.993062 17193 leader_election.cc:288] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:17:44.995779 26624 leader_election.cc:334] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:45.015198 26625 leader_election.cc:334] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:45.015354 26625 leader_election.cc:302] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:45.015597 17193 raft_consensus.cc:2726] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:17:45.016638 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:45.020790 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:45.041357 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:45.052102 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:45.052125 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:45.060002 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:17:45.085333 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:45.089623 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:17:45.108592 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:17:45.109230 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:17:45.113848 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:17:45.132087 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:17:45.161899 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:45.162068 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:17:45.178577 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:45.183706 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:45.211529 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:17:45.221489 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:17:45.231782 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:17:45.234434 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:45.280484 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:17:45.292222 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:45.326061 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:17:45.340354 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:45.403149 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:45.405479 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:45.427510 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:45.433660 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:17:45.476737 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:45.500169 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:45.528904 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:45.560654 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:45.631858 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:45.632805 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:17:45.660774 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:45.664767 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:17:45.668869 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:17:45.684456 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:17:45.707772 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:17:45.721753 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:45.745236 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:17:45.745947 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:45.746330 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:17:45.753304 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:45.776374 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:45.790418 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:17:45.872200 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:45.937963 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:45.949882 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:46.008898 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:17:46.082584 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:46.087744 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:46.149134 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:17:46.175660 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:17:46.193696 16944 raft_consensus.cc:482] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:46.193888 16944 raft_consensus.cc:504] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:17:46.194586 16944 leader_election.cc:288] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:17:46.201898 26362 leader_election.cc:334] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:46.202387 26363 leader_election.cc:334] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:46.202522 26363 leader_election.cc:302] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:46.202790 16944 raft_consensus.cc:2726] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:17:46.209115 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:17:46.212654 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:17:46.287814 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:46.328132 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:17:46.338491 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:46.338884 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:17:46.361569 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:17:46.373121 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:17:46.390987 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:46.398415 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:17:46.410586 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:46.432847 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:17:46.469278 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:17:46.488507 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:17:46.514048 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:46.555037 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:46.604600 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:46.676946 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:17:46.706782 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:17:46.708535 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:17:46.739742 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:17:46.750869 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:17:46.752650 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:46.753176 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:46.759362 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:46.772042 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:46.818114 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:46.827925 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:17:46.931842 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:17:46.973455 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:47.002344 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:17:47.004823 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:17:47.005677 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:47.027256 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:17:47.028920 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:47.039443 26755 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 144 similar messages]
W20240502 14:17:47.040928 26493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 171 similar messages]
W20240502 14:17:47.051236 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:17:47.051342 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:17:47.084740 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:17:47.096951 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:47.125238 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:17:47.160198 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:47.169864 26363 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 93 similar messages]
W20240502 14:17:47.187424 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:47.237442 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:17:47.244313 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:17:47.267788 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:17:47.269320 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:47.270303 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:47.288977 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:47.301226 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:17:47.304955 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:47.309131 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:47.322404 29112 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 79 similar messages]
I20240502 14:17:47.417078 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:17:47.419332 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:47.426405 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:17:47.441840 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:17:47.452327 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:17:47.465519 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:17:47.477530 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:17:47.478436 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:47.494728 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:47.505658 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:17:47.517597 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:17:47.554318 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:17:47.575886 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:47.627511 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:47.639554 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:47.680459 26625 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 102 similar messages]
W20240502 14:17:47.681655 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:47.682663 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:47.727339 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:17:47.739476 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:47.743582 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:47.766230 29511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240502 14:17:47.770162 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:47.791750 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:17:47.852540 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:17:47.855861 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:47.862237 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:17:47.885051 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:47.892267 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:47.901767 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:47.943030 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:17:47.959936 17284 raft_consensus.cc:482] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:47.960105 17284 raft_consensus.cc:504] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:17:47.962754 26819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cea6495041642f58e1eb5d26fc10df5" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:17:47.963008 26819 raft_consensus.cc:2454] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
W20240502 14:17:47.968034 26363 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:47.968484 17284 leader_election.cc:288] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
W20240502 14:17:47.973541 26362 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:47.973699 26362 leader_election.cc:302] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:47.974902 17284 raft_consensus.cc:2726] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:17:47.982996 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:17:47.984303 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:47.993381 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:17:47.993369 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:48.015472 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:17:48.021399 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:17:48.030009 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:48.122931 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:48.141755 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:48.195426 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:48.215538 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:48.215729 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:48.249316 17294 raft_consensus.cc:482] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:48.249507 17294 raft_consensus.cc:504] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:17:48.252111 17294 leader_election.cc:288] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
W20240502 14:17:48.257881 26493 leader_election.cc:334] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:48.259554 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:48.261504 26494 leader_election.cc:334] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:48.261646 26494 leader_election.cc:302] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:48.261919 17294 raft_consensus.cc:2726] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:17:48.312490 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:17:48.346452 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:48.350508 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:48.358402 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:48.406939 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:17:48.451974 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:17:48.479787 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:17:48.544986 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:17:48.586498 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:48.600080 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:48.618907 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:17:48.626837 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:17:48.627218 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:17:48.642962 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:48.676517 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:17:48.698298 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:48.715041 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:48.816193 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:17:48.826040 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:17:48.831241 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:17:48.885010 29222 maintenance_manager.cc:419] P 1cb67158133e4648a6eceba32d4dbba3: Scheduling FlushMRSOp(1fb1ef75829641a89c4cac539f431d80): perf score=0.033372
W20240502 14:17:48.898702 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:17:48.912711 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:48.942441 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:17:48.943673 29113 maintenance_manager.cc:643] P 1cb67158133e4648a6eceba32d4dbba3: FlushMRSOp(1fb1ef75829641a89c4cac539f431d80) complete. Timing: real 0.058s user 0.047s sys 0.007s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":6582,"dirs.run_cpu_time_us":229,"dirs.run_wall_time_us":13987,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":877,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":13568,"threads_started":2,"wal-append.queue_time_us":6706}
I20240502 14:17:48.946533 29222 maintenance_manager.cc:419] P 1cb67158133e4648a6eceba32d4dbba3: Scheduling UndoDeltaBlockGCOp(1fb1ef75829641a89c4cac539f431d80): 4586 bytes on disk
I20240502 14:17:48.947067 29113 maintenance_manager.cc:643] P 1cb67158133e4648a6eceba32d4dbba3: UndoDeltaBlockGCOp(1fb1ef75829641a89c4cac539f431d80) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
W20240502 14:17:48.972088 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:48.974712 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:17:49.050269 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:17:49.094610 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:17:49.097209 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:17:49.110338 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:49.113608 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:17:49.118702 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:17:49.119305 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:17:49.138759 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:17:49.161671 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:49.164919 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:17:49.182181 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:49.190411 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:17:49.191738 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:17:49.192641 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:49.192698 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:49.264112 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:17:49.268113 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:49.301476 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:17:49.364598 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:49.425856 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:49.436461 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:17:49.440521 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:49.568809 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:17:49.571874 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:17:49.577603 17294 raft_consensus.cc:482] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:49.577802 17294 raft_consensus.cc:504] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:17:49.578505 17294 leader_election.cc:288] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:17:49.582696 26494 leader_election.cc:334] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:49.585566 26493 leader_election.cc:334] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:49.589610 26493 leader_election.cc:302] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:49.590449 17489 raft_consensus.cc:2726] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:17:49.596822 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:49.601178 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:49.609623 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:49.628355 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:17:49.636619 16983 raft_consensus.cc:482] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:49.636826 16983 raft_consensus.cc:504] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:17:49.637533 16983 leader_election.cc:288] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:17:49.641273 26755 leader_election.cc:334] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:49.644634 26754 leader_election.cc:334] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:49.644799 26754 leader_election.cc:302] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:49.645177 16983 raft_consensus.cc:2726] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:17:49.648171 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:17:49.659729 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:17:49.670351 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:17:49.687687 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:17:49.747807 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:17:49.752651 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:49.756951 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:17:49.794263 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:49.807507 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:17:49.810022 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:49.837095 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:49.839946 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:17:49.852355 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:17:49.853858 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:17:49.866545 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:49.884064 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:49.890158 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:49.919142 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:17:49.930302 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:49.952762 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:17:50.025360 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:50.028079 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:17:50.038862 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:17:50.046968 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:17:50.068019 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:17:50.115792 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:50.142858 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:17:50.180776 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:17:50.184198 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:17:50.205011 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:50.208498 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:50.213500 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:17:50.214268 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:50.216094 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:17:50.230123 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:50.272632 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:17:50.293378 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:50.390159 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:50.404110 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:50.416793 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:50.473480 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:50.511907 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:17:50.541078 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:50.571303 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:50.584412 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:50.604269 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:50.606607 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:17:50.706144 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:17:50.712811 17729 raft_consensus.cc:482] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:50.713004 17729 raft_consensus.cc:504] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:17:50.713681 17729 leader_election.cc:288] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:17:50.716007 26624 leader_election.cc:334] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:50.717177 26625 leader_election.cc:334] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:50.717320 26625 leader_election.cc:302] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:50.717607 17729 raft_consensus.cc:2726] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:17:50.719992 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:17:50.748459 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:50.755966 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:17:50.771226 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:50.791064 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:50.816308 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:17:50.818835 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:17:50.849594 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:50.878844 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:17:50.883770 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:17:50.890712 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:17:50.941869 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:50.942704 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:51.007573 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:17:51.016537 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:51.041313 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:17:51.144357 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:51.187356 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:51.203338 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:51.257931 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:17:51.273789 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:17:51.279808 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:51.301793 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:17:51.304702 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:51.317739 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:17:51.354990 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:51.393213 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:17:51.401611 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:17:51.418975 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:17:51.445204 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:17:51.455878 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:51.469740 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:17:51.484310 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:51.500952 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:17:51.510056 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:51.524538 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:17:51.554746 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:17:51.628911 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:17:51.654182 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:51.724851 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:51.727221 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:51.745230 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:17:51.746430 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:51.808594 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:17:51.822283 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:51.852632 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:17:51.861486 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:51.863628 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:17:51.929795 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:17:51.943126 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:17:52.002542 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:17:52.011974 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:52.012029 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:17:52.016345 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:17:52.040833 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:52.045322 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:52.047207 26493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 176 similar messages]
W20240502 14:17:52.050417 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:17:52.074323 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:17:52.091960 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:17:52.094455 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:17:52.094992 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:17:52.102802 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:52.103543 26755 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240502 14:17:52.104713 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:52.115692 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:52.138821 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:52.155567 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:17:52.177484 16983 raft_consensus.cc:482] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:52.177662 16983 raft_consensus.cc:504] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:17:52.178355 16983 leader_election.cc:288] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:17:52.182425 26755 leader_election.cc:334] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:52.184194 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:52.184773 26754 leader_election.cc:334] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:52.184912 26754 leader_election.cc:302] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:52.185925 17663 raft_consensus.cc:2726] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:17:52.209066 26363 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 93 similar messages]
W20240502 14:17:52.210235 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:52.211730 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:17:52.342469 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:52.343659 29111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240502 14:17:52.408780 17663 raft_consensus.cc:482] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:52.409006 17663 raft_consensus.cc:504] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:17:52.409901 17663 leader_election.cc:288] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
W20240502 14:17:52.412559 26754 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:52.420625 26425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cea6495041642f58e1eb5d26fc10df5" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:17:52.420950 26425 raft_consensus.cc:2454] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
W20240502 14:17:52.430590 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:52.431871 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:52.432533 26755 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:52.432703 26755 leader_election.cc:302] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:52.432997 17663 raft_consensus.cc:2726] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:17:52.441550 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:17:52.446214 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:17:52.480490 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:17:52.496871 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:17:52.516146 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:52.519625 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:17:52.532526 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:17:52.576344 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:17:52.589906 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:17:52.591715 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:52.597417 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:52.625864 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:52.638727 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:17:52.645614 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:52.671358 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:17:52.680922 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:17:52.707427 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:17:52.710685 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:52.715580 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:52.751561 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:17:52.751856 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:17:52.758134 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:52.782642 29511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240502 14:17:52.874909 26625 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 105 similar messages]
W20240502 14:17:53.005087 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:53.018069 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:17:53.018206 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:53.061816 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:53.071507 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:53.071676 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:53.073848 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:17:53.090394 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:17:53.143908 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:17:53.159157 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:17:53.160053 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:53.171684 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:53.183996 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:53.187310 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:53.194501 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:17:53.195299 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:17:53.212085 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:17:53.242024 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:17:53.252485 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:53.334913 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:17:53.337400 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:17:53.339656 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:53.358937 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:17:53.436340 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:53.493295 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:53.619266 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:53.657281 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:53.661571 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:53.668512 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:17:53.672094 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:17:53.673470 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:17:53.680634 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:17:53.681954 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:17:53.730392 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:17:53.764509 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:53.798269 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:17:53.799257 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:53.815717 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:17:53.830109 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:17:53.871995 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:17:53.888790 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:17:53.906338 18144 raft_consensus.cc:482] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:53.906563 18144 raft_consensus.cc:504] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:17:53.907307 18144 leader_election.cc:288] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:17:53.919579 26493 leader_election.cc:334] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:53.919950 26494 leader_election.cc:334] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:53.920066 26494 leader_election.cc:302] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:53.920369 18144 raft_consensus.cc:2726] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:17:53.957844 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:54.023912 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:54.056895 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:54.063071 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:54.071276 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:17:54.081844 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:54.105671 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:54.125656 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:17:54.171420 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:17:54.225028 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:17:54.238487 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:17:54.254073 18144 raft_consensus.cc:482] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:54.254441 18144 raft_consensus.cc:504] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
W20240502 14:17:54.257747 26494 leader_election.cc:334] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:54.258188 18144 leader_election.cc:288] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
W20240502 14:17:54.259891 26493 leader_election.cc:334] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:54.260040 26493 leader_election.cc:302] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:54.260576 18144 raft_consensus.cc:2726] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:17:54.292246 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:17:54.323592 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:17:54.329818 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:17:54.334641 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:17:54.340152 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:17:54.357088 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:54.359228 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:17:54.360831 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:17:54.379128 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:17:54.412549 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:54.412979 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:17:54.415414 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:54.432559 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:17:54.432889 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:17:54.449893 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:17:54.451258 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:54.479995 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:54.501812 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:17:54.536728 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:54.543818 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:54.557839 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:54.564675 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:17:54.631309 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:54.638208 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:17:54.647147 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:54.652835 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:54.705161 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:17:54.716913 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:17:54.753610 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:17:54.765195 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:17:54.811878 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:54.828163 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:54.830569 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:54.867486 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:17:54.881654 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:17:54.881961 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:54.884878 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:17:54.885708 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:17:54.886624 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:17:54.891503 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:17:54.898778 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:17:54.906210 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:17:54.936201 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:54.983775 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:55.030176 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:55.048702 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:55.076280 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:55.134281 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:17:55.245854 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:17:55.292840 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:17:55.319628 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:17:55.338748 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:55.378727 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:17:55.381551 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:17:55.407557 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:17:55.412684 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:17:55.413384 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:55.418007 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:17:55.431627 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:17:55.432520 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:55.445248 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:55.451290 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:55.477710 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:55.497890 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:17:55.516726 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:55.516891 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:55.596029 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:55.646982 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:55.660181 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:55.715023 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:17:55.716507 18098 raft_consensus.cc:482] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:55.716682 18098 raft_consensus.cc:504] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:17:55.717492 18098 leader_election.cc:288] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:17:55.721822 26363 leader_election.cc:334] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:55.723174 26362 leader_election.cc:334] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:55.723300 26362 leader_election.cc:302] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:55.723708 18098 raft_consensus.cc:2726] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:17:55.796960 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:17:55.842425 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:17:55.905773 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:55.914824 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:55.920569 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:17:55.922536 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:17:55.927724 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:55.954814 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:55.959065 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:17:55.972121 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:55.991808 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:17:55.997754 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:17:56.001754 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:56.069229 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:17:56.081585 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:17:56.085933 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:17:56.114576 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:56.158491 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:17:56.173226 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:56.214146 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:17:56.216171 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:17:56.219262 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:56.346741 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:17:56.351558 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:56.376159 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:56.409159 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:17:56.462554 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:17:56.480913 18446 raft_consensus.cc:482] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:56.481154 18446 raft_consensus.cc:504] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:17:56.481926 18446 leader_election.cc:288] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:17:56.489491 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:17:56.490350 26755 leader_election.cc:334] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:56.491218 26754 leader_election.cc:334] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:56.491365 26754 leader_election.cc:302] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:56.491813 18220 raft_consensus.cc:2726] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:17:56.496884 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:17:56.499285 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:56.553614 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:17:56.596662 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:56.602155 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:17:56.610289 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:17:56.621237 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:17:56.629266 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:56.647544 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:56.653401 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:56.657157 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:17:56.669234 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:56.701395 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:56.724649 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:17:56.748208 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:17:56.821650 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:17:56.870633 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:17:56.872959 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:56.887094 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:56.904430 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:56.908895 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:56.911334 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:17:56.918696 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:17:56.978487 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:56.990520 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:57.044406 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:17:57.059684 26493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 176 similar messages]
I20240502 14:17:57.071813 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:17:57.084671 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:57.085100 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:57.088681 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:17:57.096871 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:17:57.107124 26755 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 146 similar messages]
W20240502 14:17:57.111744 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:17:57.114962 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:17:57.137265 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:57.161443 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:17:57.184060 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:57.210613 26363 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 14:17:57.231714 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:57.245429 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:57.256719 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:57.264330 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:57.270859 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:17:57.279932 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:17:57.295449 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:57.325399 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:57.362485 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:17:57.396306 29112 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 14:17:57.397887 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:57.497110 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:57.531958 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:17:57.538666 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:57.592664 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:17:57.607676 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:57.610217 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:17:57.653712 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:57.664906 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:57.678457 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:17:57.701979 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:57.714648 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:57.782433 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:57.782807 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:17:57.806988 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:57.817368 29510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240502 14:17:57.830178 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:17:57.911322 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:57.912323 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:17:57.920065 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:17:57.924077 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:17:57.958527 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:17:58.036425 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:17:58.038800 26625 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 103 similar messages]
W20240502 14:17:58.051923 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:17:58.061336 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:58.106096 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:58.151458 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:58.176254 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:58.182057 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:58.196082 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:58.220999 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:58.287648 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:58.287883 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:17:58.301333 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:58.346194 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:58.346920 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:17:58.385746 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:17:58.437793 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:17:58.450474 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:17:58.459961 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:58.478529 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:17:58.497049 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:17:58.532029 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:58.560343 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:17:58.585835 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:17:58.589955 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:17:58.599195 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:58.638752 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:17:58.662103 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:58.681883 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:17:58.694305 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:17:58.754818 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:58.760471 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:58.807435 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:58.848632 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:17:58.873350 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:17:58.931355 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:17:58.948989 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:58.951835 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:17:58.962482 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:17:58.981869 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:17:59.028055 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:17:59.033203 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:17:59.069438 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:59.080367 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:17:59.089555 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:17:59.095880 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:17:59.110788 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:17:59.122633 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:17:59.149717 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:17:59.151614 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:17:59.182067 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:59.237473 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:59.316233 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:59.324599 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:59.368672 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:17:59.378983 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:17:59.385041 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:59.392051 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:17:59.395762 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:17:59.411677 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:17:59.503746 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:17:59.523150 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:17:59.528229 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:17:59.544955 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:17:59.573578 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:17:59.625805 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:17:59.670540 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:17:59.690503 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:17:59.717576 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:59.723696 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:17:59.726094 18144 raft_consensus.cc:482] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:17:59.726274 18144 raft_consensus.cc:504] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:17:59.727108 18144 leader_election.cc:288] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:17:59.730217 26494 leader_election.cc:334] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:17:59.731698 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:17:59.732750 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:17:59.741950 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:17:59.744714 26493 leader_election.cc:334] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:17:59.744863 26493 leader_election.cc:302] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:17:59.745165 18144 raft_consensus.cc:2726] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:17:59.745249 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:17:59.771484 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:17:59.777944 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:17:59.780342 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:17:59.782583 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:17:59.784781 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:17:59.804925 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:17:59.843981 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:17:59.882839 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:17:59.913857 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:17:59.940958 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:17:59.983928 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:18:00.007189 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:18:00.111239 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:18:00.131206 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:18:00.151309 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:18:00.161494 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:18:00.169965 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:18:00.175040 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:00.194046 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:18:00.194692 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:00.232463 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:00.249011 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:00.250627 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:00.250981 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:18:00.265542 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:18:00.266616 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:00.302175 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:18:00.305207 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:18:00.313454 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:18:00.388942 18585 raft_consensus.cc:482] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:00.389155 18585 raft_consensus.cc:504] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:18:00.389803 18585 leader_election.cc:288] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:18:00.392442 26754 leader_election.cc:334] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:00.392571 26755 leader_election.cc:334] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:00.392657 26755 leader_election.cc:302] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:00.393112 18585 raft_consensus.cc:2726] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:18:00.424593 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:00.432696 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:18:00.447872 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:18:00.456543 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:18:00.465358 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:18:00.491238 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:00.494738 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:00.637706 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:18:00.651543 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:00.681329 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:18:00.706734 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:18:00.728787 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:00.742508 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:18:00.743410 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:00.767695 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:00.769719 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:00.831276 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:18:00.832314 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:00.836161 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:18:00.880949 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:00.883404 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:18:01.003912 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:01.013401 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:18:01.024199 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:18:01.072042 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:18:01.082690 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:18:01.097473 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:18:01.103089 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:01.132906 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:01.147327 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:18:01.177996 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:01.189193 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:01.189594 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:01.207494 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:01.267987 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:01.272687 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:01.295594 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:18:01.304695 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:01.312994 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:01.385614 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:18:01.414278 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:18:01.440018 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:18:01.454372 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:18:01.462347 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:18:01.505678 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:01.551563 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:01.564415 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:01.568835 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:01.630395 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:01.635682 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:01.678841 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:01.681735 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:01.705381 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:18:01.746492 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:18:01.755497 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:18:01.781956 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:01.786006 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:01.807209 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:01.832263 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:18:01.851961 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:01.854614 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:18:01.858788 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:01.861917 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:18:01.866986 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:01.886000 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:18:01.926455 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:18:01.931638 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:01.948029 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:18:01.968319 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:18:01.991011 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:02.010953 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:18:02.025529 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:02.042040 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:02.085012 26494 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 172 similar messages]
W20240502 14:18:02.118484 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:02.125377 26755 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240502 14:18:02.128610 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:18:02.218439 26363 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240502 14:18:02.237756 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:02.259963 18144 raft_consensus.cc:482] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:02.260159 18144 raft_consensus.cc:504] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:18:02.260847 18144 leader_election.cc:288] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:18:02.263099 26493 leader_election.cc:334] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:02.289528 26494 leader_election.cc:334] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:02.289716 26494 leader_election.cc:302] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:02.290077 18144 raft_consensus.cc:2726] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:18:02.291255 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:02.318660 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:18:02.362830 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:02.386785 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:02.406491 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:18:02.414732 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:02.435005 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:02.441155 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:02.447897 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:02.455861 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:18:02.461720 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:02.464561 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:18:02.486963 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:18:02.498083 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:02.531621 29112 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 14:18:02.547936 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:02.584370 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:18:02.605151 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:02.649801 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:02.659550 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:02.737295 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:18:02.746155 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:18:02.751441 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:02.752568 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:18:02.757488 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:02.773286 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:02.784586 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:18:02.808159 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:18:02.838585 29511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 138 similar messages]
I20240502 14:18:02.902930 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:02.919245 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:02.933282 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:02.935178 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:02.945376 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:18:02.946568 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:18:02.954577 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:02.964663 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:02.975557 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:18:02.976992 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:02.982522 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:18:03.037081 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:18:03.050383 26625 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240502 14:18:03.084563 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:03.108929 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:03.120321 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:18:03.151774 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:03.201458 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:03.245831 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:03.249383 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:18:03.294785 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:18:03.385552 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:18:03.489378 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:18:03.519243 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:03.561955 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:03.566519 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:03.588060 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:18:03.628075 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:03.702149 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:18:03.706540 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:18:03.717793 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:03.731125 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:03.731511 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:18:03.768409 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:03.777853 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:03.797277 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:03.803805 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:03.804759 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:18:03.822748 29511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.199:41781, user_credentials={real_user=slave}} blocked reactor thread for 120614us
W20240502 14:18:03.826928 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:18:03.833900 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:03.863303 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:18:03.910888 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:03.940251 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:03.948902 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:18:03.963536 19063 raft_consensus.cc:482] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:03.963709 19063 raft_consensus.cc:504] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:18:03.964433 19063 leader_election.cc:288] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:18:03.967007 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:03.967113 26494 leader_election.cc:334] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:03.969295 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:18:03.971709 26493 leader_election.cc:334] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:03.971817 26493 leader_election.cc:302] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:03.972069 19063 raft_consensus.cc:2726] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:18:03.986646 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:18:04.003012 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:04.100565 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:18:04.133653 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:04.166095 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:04.197414 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:18:04.203613 29621 maintenance_manager.cc:419] P 4092902b3e674857a0147f1b4f2cdb86: Scheduling FlushMRSOp(aedd72fd3aeb48d68f26a042069cebab): perf score=0.033399
W20240502 14:18:04.221311 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:18:04.222193 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:04.241709 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:04.267042 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:04.268704 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:04.287497 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:18:04.328091 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:18:04.330194 29512 maintenance_manager.cc:643] P 4092902b3e674857a0147f1b4f2cdb86: FlushMRSOp(aedd72fd3aeb48d68f26a042069cebab) complete. Timing: real 0.125s user 0.047s sys 0.025s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":19786,"dirs.run_cpu_time_us":209,"dirs.run_wall_time_us":1801,"drs_written":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1286,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":26386,"threads_started":2,"wal-append.queue_time_us":6513}
I20240502 14:18:04.333971 29621 maintenance_manager.cc:419] P 4092902b3e674857a0147f1b4f2cdb86: Scheduling UndoDeltaBlockGCOp(aedd72fd3aeb48d68f26a042069cebab): 4586 bytes on disk
I20240502 14:18:04.335201 29512 maintenance_manager.cc:643] P 4092902b3e674857a0147f1b4f2cdb86: UndoDeltaBlockGCOp(aedd72fd3aeb48d68f26a042069cebab) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":127,"lbm_reads_lt_1ms":4}
W20240502 14:18:04.339048 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:18:04.354913 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:04.392910 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:04.429706 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:18:04.430058 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:04.466185 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:04.496187 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:04.496838 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:18:04.501511 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:04.519694 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:18:04.522820 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:04.532274 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:04.538170 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:04.553085 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:18:04.593029 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:18:04.593520 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:18:04.664695 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:04.667804 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:04.668807 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:04.704934 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:04.852591 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:18:04.861088 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:04.870049 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:18:04.878028 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:04.924917 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:18:04.945999 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:04.948966 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:04.958369 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:04.988118 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:05.032919 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:18:05.042848 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:05.047240 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:18:05.048483 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:18:05.049901 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:18:05.050025 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:18:05.060536 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:18:05.122270 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:05.131052 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:05.262364 18723 raft_consensus.cc:482] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:05.262568 18723 raft_consensus.cc:504] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:18:05.263265 18723 leader_election.cc:288] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:18:05.266359 26362 leader_election.cc:334] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:05.266497 26363 leader_election.cc:334] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:05.266604 26363 leader_election.cc:302] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:05.266878 18723 raft_consensus.cc:2726] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:18:05.277536 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:05.281824 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:05.309528 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:05.310958 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:18:05.374109 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:18:05.444696 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:18:05.455073 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:05.472566 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:18:05.477751 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:05.478812 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:05.493218 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:18:05.507483 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:05.530479 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:18:05.531224 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:05.586274 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:05.606079 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:18:05.655567 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:18:05.657069 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:05.663259 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:05.670269 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:05.671464 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:05.695681 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:05.708078 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:05.719941 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:18:05.755506 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:18:05.800014 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:18:05.843909 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:05.855051 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:05.875964 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:05.931703 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:18:05.956346 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:18:05.983911 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:06.017094 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:18:06.037637 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:06.052027 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:06.055599 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:06.056459 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:06.077347 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:18:06.081714 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:06.082698 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:18:06.104470 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:18:06.183001 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:06.191042 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:06.221843 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:18:06.225349 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:06.257622 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:18:06.294641 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:18:06.316176 19443 raft_consensus.cc:482] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:06.316376 19443 raft_consensus.cc:504] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:18:06.317171 19443 leader_election.cc:288] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:18:06.318921 26754 leader_election.cc:334] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:06.319095 26755 leader_election.cc:334] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:06.319180 26755 leader_election.cc:302] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:06.319456 19443 raft_consensus.cc:2726] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:18:06.352717 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:18:06.436304 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:18:06.437629 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:18:06.476130 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:06.480985 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:06.488288 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:06.528795 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:06.551841 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:06.587512 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:18:06.621924 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:06.652699 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:06.663501 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:18:06.694716 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:06.697521 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:18:06.701766 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:06.703411 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:06.724961 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:06.726326 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:06.727218 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:06.791667 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:06.797174 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:18:06.804968 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:06.809438 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:18:06.828099 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:18:06.850687 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:06.857175 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:06.884536 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:06.928140 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:06.976245 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:06.990733 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:18:07.012171 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:07.059537 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:18:07.072113 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:07.096310 26493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 176 similar messages]
W20240502 14:18:07.128173 26755 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240502 14:18:07.156478 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:18:07.162101 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:07.165417 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:18:07.182324 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:07.183969 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:07.205797 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:07.220291 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:18:07.221158 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:07.235590 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:18:07.243197 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:18:07.243686 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:07.253217 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:18:07.310366 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:07.315366 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:07.317735 26362 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240502 14:18:07.368094 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:18:07.387564 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:07.434815 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:18:07.440240 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:07.449551 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:07.464228 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:07.483714 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:18:07.556319 29111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 14:18:07.566766 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:18:07.572302 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:18:07.574960 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:07.689853 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:07.706951 19443 raft_consensus.cc:482] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:07.707162 19443 raft_consensus.cc:504] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:18:07.708091 19443 leader_election.cc:288] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:18:07.709424 26427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cea6495041642f58e1eb5d26fc10df5" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:18:07.709666 26427 raft_consensus.cc:2454] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
W20240502 14:18:07.711288 26755 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:07.711776 26754 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:07.711879 26754 leader_election.cc:302] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:07.712353 19443 raft_consensus.cc:2726] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:18:07.725435 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:07.782630 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:07.793560 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:07.793934 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:07.800848 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:07.830819 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:07.844599 29511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 137 similar messages]
W20240502 14:18:07.850571 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:07.855091 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:18:07.875887 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:18:07.879928 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:18:07.923615 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:07.934634 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:18:07.960903 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:18:08.018864 19443 raft_consensus.cc:482] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:08.019125 19443 raft_consensus.cc:504] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:18:08.020884 19443 leader_election.cc:288] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:18:08.021786 26754 leader_election.cc:334] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:08.023020 26755 leader_election.cc:334] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:08.023133 26755 leader_election.cc:302] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:08.023429 19443 raft_consensus.cc:2726] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:18:08.036628 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:08.065966 18723 raft_consensus.cc:482] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:08.066171 18723 raft_consensus.cc:504] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:18:08.067307 18723 leader_election.cc:288] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:18:08.068962 26362 leader_election.cc:334] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:08.069828 26363 leader_election.cc:334] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:08.069937 26363 leader_election.cc:302] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:08.070374 19594 raft_consensus.cc:2726] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:18:08.078743 26625 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 97 similar messages]
W20240502 14:18:08.080322 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:18:08.109766 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:08.129945 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:08.283919 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:08.293541 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:18:08.297194 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:18:08.300282 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:08.325815 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:08.343250 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:08.344981 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:08.351171 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:18:08.373574 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:08.391953 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:18:08.423632 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:08.425100 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:08.425441 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:08.427083 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:08.445704 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:18:08.446789 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:08.511358 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:08.531939 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:18:08.536939 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:18:08.541325 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:18:08.598196 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:08.626749 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:18:08.640453 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:18:08.665349 29621 maintenance_manager.cc:419] P 4092902b3e674857a0147f1b4f2cdb86: Scheduling FlushMRSOp(0290ebbcf609427cae8fce6637019336): perf score=0.033354
I20240502 14:18:08.704699 29512 maintenance_manager.cc:643] P 4092902b3e674857a0147f1b4f2cdb86: FlushMRSOp(0290ebbcf609427cae8fce6637019336) complete. Timing: real 0.039s user 0.033s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":806,"dirs.run_cpu_time_us":181,"dirs.run_wall_time_us":1658,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":717,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":980,"threads_started":2,"wal-append.queue_time_us":360}
I20240502 14:18:08.706784 29621 maintenance_manager.cc:419] P 4092902b3e674857a0147f1b4f2cdb86: Scheduling UndoDeltaBlockGCOp(0290ebbcf609427cae8fce6637019336): 4840 bytes on disk
I20240502 14:18:08.707655 29512 maintenance_manager.cc:643] P 4092902b3e674857a0147f1b4f2cdb86: UndoDeltaBlockGCOp(0290ebbcf609427cae8fce6637019336) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":225,"lbm_reads_lt_1ms":4}
W20240502 14:18:08.711614 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:18:08.751618 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:08.786943 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:18:08.796444 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:18:08.815901 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:18:08.832944 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:18:08.846662 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:08.922616 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:18:08.923787 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:08.955761 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:08.957491 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:08.979735 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:18:08.989470 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:08.990307 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:09.036895 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:09.037609 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:18:09.038738 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:09.049937 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:09.070683 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:09.071475 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:09.084733 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:09.099232 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:18:09.100241 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:18:09.115267 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:09.118582 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:09.126274 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:18:09.153649 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:18:09.252030 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:09.269014 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:09.428468 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:18:09.430824 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:09.453886 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:09.490581 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:09.534150 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:09.536031 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:18:09.590593 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:09.612907 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:09.614549 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:18:09.624485 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:09.656606 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:09.692351 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:09.715744 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:18:09.734862 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:09.755805 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:18:09.756829 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:18:09.759632 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:18:09.763857 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:09.763856 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:09.765686 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:09.765980 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:18:09.797755 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:09.797864 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:09.871429 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:09.879374 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:09.884177 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:09.921953 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:09.959347 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:18:09.976943 19937 raft_consensus.cc:482] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:09.977250 19937 raft_consensus.cc:504] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:18:09.978029 19937 leader_election.cc:288] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:18:09.979610 26625 leader_election.cc:334] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:09.979820 26624 leader_election.cc:334] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:09.979929 26624 leader_election.cc:302] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:09.980314 19937 raft_consensus.cc:2726] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:18:09.980909 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:10.006673 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:18:10.017825 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:10.075065 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:18:10.076850 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:18:10.105156 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:10.154341 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:10.171451 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:18:10.238917 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:10.288646 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:18:10.293246 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:10.338739 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:10.342283 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:18:10.345337 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:10.352665 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:10.358995 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:18:10.368160 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:18:10.369311 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:10.378278 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:18:10.381173 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:18:10.394085 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:18:10.504859 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:10.584383 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:18:10.591225 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:10.634982 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:10.637720 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:10.664350 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:10.667155 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:18:10.795187 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:18:10.811621 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:10.827795 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:10.833667 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:18:10.837972 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:10.848021 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:10.859761 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:18:10.861225 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:10.877570 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:18:10.884311 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:10.897505 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:18:10.910873 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:18:10.940884 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:10.996088 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:11.043987 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:11.129465 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:11.140197 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:18:11.166916 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:11.191771 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:18:11.206266 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:18:11.209969 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:18:11.270486 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:11.324731 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:18:11.327492 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:11.344496 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:11.345060 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:18:11.346505 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:11.348817 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:11.362882 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:18:11.394887 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:18:11.406625 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:11.441653 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:18:11.445116 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:18:11.475154 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:11.497871 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:11.507917 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:11.508037 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:11.527369 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:11.576123 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:11.653367 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:18:11.663094 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:11.705139 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:18:11.707918 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:18:11.711740 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:11.751983 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:18:11.792059 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:18:11.794512 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:18:11.830463 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:11.834545 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:11.847904 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:18:11.850646 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:11.868759 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:11.893921 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:11.905552 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:11.922947 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:11.981583 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:12.004691 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:12.059552 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:18:12.060042 20219 raft_consensus.cc:482] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:12.060226 20219 raft_consensus.cc:504] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:18:12.060941 20219 leader_election.cc:288] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:18:12.063916 26624 leader_election.cc:334] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:12.064474 26625 leader_election.cc:334] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:12.064585 26625 leader_election.cc:302] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:12.064811 19937 raft_consensus.cc:2726] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:18:12.077188 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:12.090616 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:12.091146 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:12.099478 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:12.129910 26754 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 136 similar messages]
W20240502 14:18:12.163959 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:12.166358 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:18:12.178428 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:12.193163 26493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 170 similar messages]
W20240502 14:18:12.194424 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:12.200973 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:18:12.218956 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:18:12.290069 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:18:12.322026 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:18:12.352806 26362 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 90 similar messages]
I20240502 14:18:12.430038 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:12.444380 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:18:12.451577 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:18:12.451661 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:18:12.461827 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:12.462972 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:12.508235 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:18:12.516296 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:12.516957 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:12.556224 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:12.567147 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:18:12.574112 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:18:12.577051 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:12.577528 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:12.584905 29111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 14:18:12.591110 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:12.594853 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:18:12.595959 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:12.601683 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:18:12.676973 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:12.782750 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:18:12.796598 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:12.802590 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:18:12.824254 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:18:12.847362 29511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240502 14:18:12.848649 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:18:12.874411 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:18:12.884791 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:18:12.914078 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:13.012845 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:13.036439 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:13.038209 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:13.048693 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:18:13.063004 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:13.070364 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:13.072034 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:13.085973 26624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 105 similar messages]
I20240502 14:18:13.091081 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:13.097102 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:13.115474 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:18:13.175158 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:18:13.278287 20207 raft_consensus.cc:482] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:13.278483 20207 raft_consensus.cc:504] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:18:13.279253 20207 leader_election.cc:288] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:18:13.280550 26754 leader_election.cc:334] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:13.280755 26755 leader_election.cc:334] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:13.280861 26755 leader_election.cc:302] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:13.281087 20207 raft_consensus.cc:2726] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:18:13.285154 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:18:13.304916 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:18:13.351387 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:13.392613 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:18:13.405151 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:18:13.411248 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:18:13.471168 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:13.501703 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:13.505017 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:13.504096 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:13.506695 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:18:13.556365 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:13.557324 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:13.579470 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:13.588783 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:18:13.593154 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:18:13.599764 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:13.600014 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:18:13.632508 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:13.640767 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:13.675660 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:13.737980 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:13.777597 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:18:13.800062 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:13.801903 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:18:13.830297 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:18:13.833823 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:13.851686 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:18:13.875651 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:18:13.922179 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:18:13.957480 26274 maintenance_manager.cc:419] P 09bae9f1528d44c09ade01fa361cebd0: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033401
W20240502 14:18:13.972653 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:13.973026 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:13.979475 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:18:13.980538 26273 maintenance_manager.cc:643] P 09bae9f1528d44c09ade01fa361cebd0: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.022s user 0.009s sys 0.000s Metrics: {"bytes_written":6458,"delete_count":0,"lbm_write_time_us":353,"lbm_writes_lt_1ms":5,"reinsert_count":0,"thread_start_us":7837,"threads_started":1,"update_count":104,"wal-append.queue_time_us":431}
W20240502 14:18:14.030894 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:18:14.056113 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:14.084316 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:14.093322 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:14.102667 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:18:14.106514 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:14.112777 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:18:14.118029 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:18:14.119187 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:14.121151 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:14.123289 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:14.138510 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:18:14.159235 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:14.246903 19896 raft_consensus.cc:482] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:14.247115 19896 raft_consensus.cc:504] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:18:14.247895 19896 leader_election.cc:288] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
W20240502 14:18:14.255995 26494 leader_election.cc:334] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:14.279843 26493 leader_election.cc:334] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:14.279985 26493 leader_election.cc:302] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:14.281900 20448 raft_consensus.cc:2726] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:18:14.316172 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:14.329216 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:18:14.330536 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:14.415145 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:14.416749 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:14.467670 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:14.474802 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:18:14.535950 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:14.554853 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:18:14.573313 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:14.590874 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:14.595896 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:18:14.610796 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:14.639178 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:14.684603 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:18:14.689956 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:14.709849 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:18:14.738180 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:14.745764 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:14.748740 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:14.752107 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:14.761210 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:18:14.777720 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:18:14.779511 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:14.779727 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:18:14.780411 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:14.798815 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:14.801297 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:18:14.804816 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:14.840190 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:18:14.888955 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:18:14.955911 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:14.979413 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:18:14.991081 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:15.030282 20522 raft_consensus.cc:482] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:15.030514 20522 raft_consensus.cc:504] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:18:15.031442 20522 leader_election.cc:288] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
W20240502 14:18:15.031733 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:15.032872 26363 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:15.032770 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cea6495041642f58e1eb5d26fc10df5" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:18:15.033051 26818 raft_consensus.cc:2454] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
W20240502 14:18:15.033494 26362 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:15.033614 26362 leader_election.cc:302] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:15.033881 20522 raft_consensus.cc:2726] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:18:15.053107 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:15.121953 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:18:15.150991 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:18:15.166003 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:15.182348 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:18:15.254873 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:15.278524 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:15.378211 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:18:15.396103 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:18:15.451459 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:15.461386 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:18:15.470530 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:18:15.538776 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:15.542984 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:18:15.568511 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:15.602831 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:15.604892 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:18:15.639708 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:15.639858 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:15.652334 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:15.652470 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:18:15.662904 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:15.734598 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:18:15.748679 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:15.839560 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:15.851454 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:15.880848 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:18:15.887712 20616 raft_consensus.cc:482] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:15.887986 20616 raft_consensus.cc:504] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:18:15.891454 20601 raft_consensus.cc:482] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:15.891656 20601 raft_consensus.cc:504] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
W20240502 14:18:15.892042 26494 leader_election.cc:334] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:15.892308 20616 leader_election.cc:288] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
I20240502 14:18:15.892488 20601 leader_election.cc:288] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:18:15.896247 26493 leader_election.cc:334] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:15.896437 26493 leader_election.cc:302] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:15.896777 20616 raft_consensus.cc:2726] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:18:15.900740 26755 leader_election.cc:334] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:15.901393 26754 leader_election.cc:334] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:15.901495 26754 leader_election.cc:302] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:15.902068 19174 raft_consensus.cc:2726] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:18:15.927242 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:18:15.953528 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:18:15.968163 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:18:16.005323 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:16.010129 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:18:16.049015 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:16.050089 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:16.075563 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:16.076629 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:18:16.085216 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:16.105320 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:16.124589 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:18:16.125612 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:16.147346 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:16.175662 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:18:16.184798 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:18:16.193393 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:18:16.207656 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:16.222669 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:18:16.238467 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:18:16.293460 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:16.298727 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:16.300747 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:16.339504 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:16.342917 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:18:16.376603 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:18:16.410708 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:18:16.439934 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:18:16.499032 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:16.515427 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:18:16.526970 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:16.530400 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:18:16.549041 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:16.566267 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:18:16.597823 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:16.614571 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:16.617542 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:16.654234 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:18:16.685156 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:16.710515 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:16.731353 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:16.736809 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:18:16.738165 20616 raft_consensus.cc:482] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:16.738317 20616 raft_consensus.cc:504] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:18:16.738931 20616 leader_election.cc:288] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:18:16.740592 26493 leader_election.cc:334] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:16.740773 26494 leader_election.cc:334] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:16.740883 26494 leader_election.cc:302] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:16.741122 20616 raft_consensus.cc:2726] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:18:16.908802 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:16.919948 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:18:16.931504 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:16.942049 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:18:17.006304 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:17.009063 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:18:17.014986 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:17.016124 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:17.031836 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:17.037739 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:18:17.042133 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:18:17.065327 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:17.069139 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:18:17.075978 20207 raft_consensus.cc:482] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:17.076159 20207 raft_consensus.cc:504] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:18:17.076972 20207 leader_election.cc:288] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:18:17.078763 26427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cea6495041642f58e1eb5d26fc10df5" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:18:17.078991 26427 raft_consensus.cc:2454] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
W20240502 14:18:17.081317 26755 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:17.083585 26754 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:17.083693 26754 leader_election.cc:302] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:17.084770 20207 raft_consensus.cc:2726] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:18:17.086881 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:18:17.088594 20616 raft_consensus.cc:482] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:17.088750 20616 raft_consensus.cc:504] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:18:17.089463 20616 leader_election.cc:288] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
W20240502 14:18:17.107510 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:18:17.114446 26494 leader_election.cc:334] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:17.124192 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:17.124326 26493 leader_election.cc:334] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:17.124431 26493 leader_election.cc:302] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:17.124663 19896 raft_consensus.cc:2726] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:18:17.126549 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:18:17.130115 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:17.157218 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:17.182979 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:17.218529 26754 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 151 similar messages]
W20240502 14:18:17.222414 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:18:17.250836 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:17.258234 26493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 180 similar messages]
W20240502 14:18:17.259925 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:17.271137 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:18:17.316382 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:18:17.325037 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:18:17.333650 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:18:17.337996 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:18:17.384161 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:17.390914 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:18:17.410315 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:18:17.437387 26362 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 14:18:17.488473 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:17.488925 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:17.505647 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:18:17.505642 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:17.507879 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:18:17.520792 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:17.521600 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:17.572885 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:17.589495 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:17.597774 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:17.614051 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:17.614849 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:17.627198 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:17.635763 29111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 14:18:17.705346 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:18:17.705665 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:18:17.720563 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:17.770898 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:17.814940 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:18:17.881203 20207 raft_consensus.cc:482] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:17.881408 20207 raft_consensus.cc:504] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:18:17.882126 20207 leader_election.cc:288] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:18:17.883700 26755 leader_election.cc:334] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:17.888778 26754 leader_election.cc:334] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:17.888933 26754 leader_election.cc:302] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:17.889500 19174 raft_consensus.cc:2726] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:18:17.943621 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:17.951181 29511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 138 similar messages]
W20240502 14:18:17.955972 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:18:17.962296 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:18.000795 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:18:18.003535 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:18.008147 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:18:18.009402 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:18.035830 20522 raft_consensus.cc:482] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:18.036033 20522 raft_consensus.cc:504] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:18:18.036785 20522 leader_election.cc:288] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:18:18.039736 26363 leader_election.cc:334] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:18.041550 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:18.041891 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:18.050520 26362 leader_election.cc:334] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:18.050665 26362 leader_election.cc:302] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:18.051177 20522 raft_consensus.cc:2726] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:18:18.081102 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:18.124666 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:18.125453 26625 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240502 14:18:18.137789 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:18:18.163009 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:18:18.172325 19896 raft_consensus.cc:482] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:18.172544 19896 raft_consensus.cc:504] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:18:18.173147 19896 leader_election.cc:288] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:18:18.174259 26493 leader_election.cc:334] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:18.188489 26494 leader_election.cc:334] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:18.188665 26494 leader_election.cc:302] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:18.189702 20863 raft_consensus.cc:2726] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:18:18.192979 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:18.213737 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:18:18.241452 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:18:18.247041 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:18.319607 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:18.422637 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:18:18.428331 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:18.433948 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:18:18.436101 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:18.446926 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:18:18.513531 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:18.528520 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:18:18.541375 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:18:18.553825 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:18:18.581523 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:18.614811 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:18:18.624243 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:18.634977 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:18.640326 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:18:18.642525 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:18.666615 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:18.674160 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:18.684577 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:18.715329 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:18.715900 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:18:18.724697 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:18.729015 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:18.749593 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:18.791186 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:18:18.832469 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:18:18.856842 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:18.892634 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:18.905086 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:18:18.925282 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:18:19.055930 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:19.064160 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:19.087612 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:19.098544 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:19.134050 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:19.137918 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:19.167102 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:19.204701 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:19.226893 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:19.234930 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:19.235602 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:18:19.236858 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:19.310055 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:19.311880 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:19.330011 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:18:19.334336 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:19.338515 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:18:19.402640 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:19.464349 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:19.476105 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:18:19.497813 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:18:19.525923 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:18:19.527328 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:18:19.540850 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:19.567483 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:19.602944 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:19.606089 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:19.656842 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:19.657999 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:18:19.682241 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:19.717823 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:19.721813 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:18:19.724314 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:19.752359 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:18:19.754040 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:19.766312 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:18:19.804618 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:18:19.804822 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:18:19.810222 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:19.812431 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:19.824105 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:19.858655 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:19.870890 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:18:19.881260 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:19.886113 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:18:19.890389 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:19.955904 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:18:20.040922 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:20.057121 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:18:20.065042 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:20.102355 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:20.127827 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:20.144722 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:20.241706 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:20.295429 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:20.335247 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:20.335572 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:18:20.349720 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:18:20.355898 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:20.374039 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:20.401916 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:20.419921 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:18:20.431460 20207 raft_consensus.cc:482] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:20.431632 20207 raft_consensus.cc:504] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:18:20.432240 20207 leader_election.cc:288] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:18:20.435002 26754 leader_election.cc:334] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:20.437294 26755 leader_election.cc:334] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:20.437407 26755 leader_election.cc:302] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:20.437625 20207 raft_consensus.cc:2726] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:18:20.492743 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:20.501695 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:18:20.532912 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:20.599854 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:20.600160 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:20.603904 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:18:20.610742 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:18:20.638250 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:18:20.666126 19896 raft_consensus.cc:482] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:20.666285 19896 raft_consensus.cc:504] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:18:20.666882 19896 leader_election.cc:288] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:18:20.668330 26494 leader_election.cc:334] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:20.668506 26493 leader_election.cc:334] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:20.668603 26493 leader_election.cc:302] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:20.668838 19896 raft_consensus.cc:2726] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:18:20.708207 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:20.721257 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:20.765044 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:18:20.784477 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:20.868130 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:18:20.959192 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:20.966666 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:18:20.981231 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:18:21.034860 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:18:21.037148 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:18:21.049706 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:21.063210 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:18:21.069950 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:21.074538 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:21.100492 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:21.108906 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:21.119055 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:21.133723 29510 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.197:38715, user_credentials={real_user=slave}} blocked reactor thread for 65925.9us
I20240502 14:18:21.144367 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:21.147609 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:18:21.154590 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:21.190230 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:21.195596 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:21.204106 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:21.214444 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:18:21.242151 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:18:21.263746 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:18:21.286131 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:18:21.318447 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:21.329536 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:21.378728 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:21.395005 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:18:21.446866 19896 raft_consensus.cc:482] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:21.447062 19896 raft_consensus.cc:504] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:18:21.447705 19896 leader_election.cc:288] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
W20240502 14:18:21.449522 26494 leader_election.cc:334] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:21.451103 26493 leader_election.cc:334] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:21.451223 26493 leader_election.cc:302] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:21.451725 20616 raft_consensus.cc:2726] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:18:21.474829 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:18:21.478549 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:21.503973 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:18:21.555209 21130 raft_consensus.cc:482] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:21.555408 21130 raft_consensus.cc:504] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:18:21.556133 21130 leader_election.cc:288] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:18:21.563292 26363 leader_election.cc:334] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:21.568051 26362 leader_election.cc:334] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:21.569063 26362 leader_election.cc:302] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:21.569429 21130 raft_consensus.cc:2726] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:18:21.584523 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:18:21.598223 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:18:21.619520 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:21.625483 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:21.636459 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:18:21.640241 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:21.684365 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:21.691932 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:21.695945 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:21.702831 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:21.718451 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:21.732038 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:18:21.745548 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:21.759639 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:18:21.760744 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:21.776029 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:21.798666 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:21.836841 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:21.895465 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:21.902396 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:21.972978 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:18:22.014981 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:18:22.037089 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:18:22.049026 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:18:22.052775 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:22.053980 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:22.077649 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:18:22.079777 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:22.103559 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:18:22.117717 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:22.133775 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:18:22.139467 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:22.141629 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:18:22.159778 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:22.180593 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:18:22.262053 26493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 173 similar messages]
W20240502 14:18:22.272222 26755 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 143 similar messages]
W20240502 14:18:22.279769 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:22.281221 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:18:22.292327 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:22.294674 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:22.304958 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:22.314019 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:22.318603 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:22.356189 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:18:22.387632 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:22.402819 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:22.481298 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:18:22.492434 26362 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 94 similar messages]
W20240502 14:18:22.511335 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:18:22.570431 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:18:22.581179 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:18:22.613001 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:18:22.614323 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:22.624727 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:22.645241 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:22.648902 29111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 78 similar messages]
I20240502 14:18:22.666467 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:18:22.673610 20207 raft_consensus.cc:482] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:22.673800 20207 raft_consensus.cc:504] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:18:22.674681 20207 leader_election.cc:288] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:18:22.675351 26427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cea6495041642f58e1eb5d26fc10df5" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:18:22.675585 26427 raft_consensus.cc:2454] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
W20240502 14:18:22.677455 26754 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:22.679937 26755 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:22.680150 26755 leader_election.cc:302] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:22.680495 21034 raft_consensus.cc:2726] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:18:22.701126 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:22.718888 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:22.741227 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:18:22.748723 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:22.759197 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:22.762650 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:22.771919 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:22.846266 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:22.872888 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:22.978667 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:18:22.993604 29510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 144 similar messages]
W20240502 14:18:22.996888 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:18:23.056196 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:23.076084 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:23.085803 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:18:23.090668 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:23.092000 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:18:23.097162 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:18:23.099711 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:18:23.104010 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:23.108022 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:23.160601 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:18:23.160766 26625 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240502 14:18:23.180215 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:23.187705 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:23.203004 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:23.236126 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:18:23.428174 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:18:23.429755 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:23.461860 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:18:23.475600 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:18:23.509898 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:23.520896 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:18:23.523478 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:23.541711 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:23.543885 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:18:23.570724 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:18:23.572127 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:23.610037 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:23.638895 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:23.653947 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:23.655182 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:23.664937 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:23.666702 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:23.674046 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:23.685448 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:18:23.689868 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:23.690215 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:18:23.778875 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:18:23.789232 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:18:23.818681 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:18:23.842630 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:23.860656 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:18:23.862286 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:18:23.876986 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:23.894547 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:23.922250 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:18:23.989146 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:18:24.028249 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:24.043946 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:24.056406 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:24.058238 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:18:24.058262 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:18:24.068253 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:24.079105 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:24.088950 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:24.107736 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:24.153847 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:24.199378 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:18:24.209517 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:24.214804 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:24.239526 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:24.243299 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:18:24.257619 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:24.266400 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:24.344751 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:24.344753 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:24.414973 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:18:24.424577 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:24.489213 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:24.493121 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:18:24.520895 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:18:24.539634 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:24.542131 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:18:24.574425 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:24.578919 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:24.601346 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:24.609709 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:18:24.621147 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:18:24.646525 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:24.666491 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:24.685506 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:24.709327 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:24.719203 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:18:24.729799 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:18:24.759377 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:18:24.759605 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:24.772820 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:18:24.781256 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:24.788381 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:18:24.796375 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:24.799594 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:24.815277 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:24.873847 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:24.919612 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:18:25.003720 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:18:25.051990 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:25.065239 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:18:25.105553 21563 raft_consensus.cc:482] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:25.105791 21563 raft_consensus.cc:504] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:18:25.106530 21563 leader_election.cc:288] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:18:25.110077 26624 leader_election.cc:334] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:25.114318 26625 leader_election.cc:334] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:25.114465 26625 leader_election.cc:302] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:25.115183 21563 raft_consensus.cc:2726] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:18:25.130506 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:25.221014 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:18:25.230064 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:25.236579 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:18:25.243273 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:18:25.246773 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:25.252383 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:25.318004 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:25.367074 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:25.367322 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:25.390237 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:18:25.406145 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:18:25.441856 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:25.443318 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:18:25.523211 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:25.526939 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:25.589320 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:25.609745 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:25.610411 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:18:25.689097 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:18:25.731842 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:25.752658 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:25.761497 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:18:25.766535 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:18:25.780800 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:25.809960 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:18:25.853931 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:25.854535 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:25.858282 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:25.867378 26754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.194:36025, user_credentials={real_user=slave}} blocked reactor thread for 64603.8us
W20240502 14:18:25.891073 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:25.921561 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:18:25.925153 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:25.961684 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:25.962061 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:25.990989 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:26.043543 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:18:26.057226 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:18:26.078421 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:26.126861 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:26.141737 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:26.143235 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:18:26.200373 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:18:26.266404 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:18:26.269078 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:18:26.276145 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:18:26.285310 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:26.364624 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:26.374089 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:18:26.388895 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:26.393160 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:26.431185 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:26.442360 26362 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.197:38715, user_credentials={real_user=slave}} blocked reactor thread for 63797.3us
W20240502 14:18:26.443693 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:26.458465 26363 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.200:46097, user_credentials={real_user=slave}} blocked reactor thread for 81968.8us
W20240502 14:18:26.463914 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:26.477195 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:26.502558 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:26.551383 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:18:26.553397 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:26.576853 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:26.585371 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:18:26.609690 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:26.615733 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:26.615859 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:26.632061 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:18:26.636771 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:26.651710 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:26.654047 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:18:26.675690 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:18:26.677294 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:26.690498 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:26.809254 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:26.814357 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:26.866653 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:26.868343 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:18:26.877305 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:26.888379 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:26.922264 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:26.937554 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:26.947957 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:26.954216 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:26.973655 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:27.025302 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:18:27.064787 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:27.071378 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:27.084966 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:27.109961 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:18:27.112144 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:27.115481 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:18:27.125669 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:18:27.137607 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:27.142297 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:27.224669 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:27.253151 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:27.299382 26754 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 143 similar messages]
W20240502 14:18:27.301540 26493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 169 similar messages]
W20240502 14:18:27.304348 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:27.304633 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:27.351365 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:27.382784 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:27.384531 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:27.396422 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:27.398000 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:27.403555 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:27.485536 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:27.495922 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:27.513268 26362 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 87 similar messages]
W20240502 14:18:27.513888 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:27.573062 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:18:27.612715 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:27.613524 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:27.644285 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:27.663221 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:27.664564 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:27.681525 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:27.734771 29111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240502 14:18:27.831693 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:27.849714 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:18:27.868700 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:27.916530 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:27.944522 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:27.951754 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:27.957995 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:27.983184 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:28.021353 29511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 141 similar messages]
I20240502 14:18:28.043262 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:28.049278 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:18:28.054579 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:28.057307 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:18:28.057719 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:28.060504 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:18:28.068627 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:28.074054 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:28.078438 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:28.090373 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:28.181001 26625 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 102 similar messages]
W20240502 14:18:28.192152 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:28.192591 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:28.273969 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:18:28.335005 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:18:28.352803 20207 raft_consensus.cc:482] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:28.352999 20207 raft_consensus.cc:504] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:18:28.354000 20207 leader_election.cc:288] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
W20240502 14:18:28.356251 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:28.356604 26755 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:28.358026 26425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cea6495041642f58e1eb5d26fc10df5" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:18:28.358268 26425 raft_consensus.cc:2454] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
W20240502 14:18:28.362473 26754 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:28.362604 26754 leader_election.cc:302] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:28.362845 20207 raft_consensus.cc:2726] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:18:28.397894 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:18:28.430054 21706 raft_consensus.cc:482] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:28.430256 21706 raft_consensus.cc:504] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:18:28.430903 21706 leader_election.cc:288] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
W20240502 14:18:28.433163 26494 leader_election.cc:334] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:28.436970 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:28.442222 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:28.449666 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:28.462116 26493 leader_election.cc:334] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:28.462255 26493 leader_election.cc:302] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:28.462487 21465 raft_consensus.cc:2726] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:18:28.499653 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:28.525154 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:28.525432 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:28.541750 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:18:28.555411 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:28.577256 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:28.603313 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:28.650094 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:18:28.655447 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:18:28.655433 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:28.665328 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:28.668215 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:28.674306 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:18:28.686926 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:28.701957 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:28.702904 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:28.715901 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:28.728381 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:28.760372 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:28.813041 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:28.843297 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:28.899154 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:18:28.905529 21942 raft_consensus.cc:482] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:28.905714 21942 raft_consensus.cc:504] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:18:28.906428 21942 leader_election.cc:288] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:18:28.910319 26363 leader_election.cc:334] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:28.910495 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:28.911633 26362 leader_election.cc:334] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:28.911761 26362 leader_election.cc:302] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:28.912024 21759 raft_consensus.cc:2726] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:18:28.922394 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:28.981175 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:28.996376 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:29.009816 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:29.022365 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:29.038837 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:29.049459 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:29.106050 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:29.106995 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:18:29.116562 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:29.121743 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:29.146870 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:18:29.162707 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:29.165498 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:29.166530 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:29.173188 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:18:29.174626 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:29.199187 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:29.304359 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:29.361343 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:29.383512 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:29.387975 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:29.413408 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:29.438230 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:29.445101 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:29.468744 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:29.478143 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:29.513609 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:29.570483 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:29.577158 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:29.596431 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:29.625357 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:29.628300 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:29.665303 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:18:29.684629 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:29.697288 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:18:29.708866 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:29.711570 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:29.723114 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:29.723831 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:18:29.742195 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:29.773928 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:29.783160 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:29.788059 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:29.789851 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:18:29.799422 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:29.825758 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:29.860013 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:29.932422 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:29.969874 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:30.089684 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:30.128162 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:18:30.181528 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:30.182890 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:18:30.205587 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:30.206425 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:30.206970 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:18:30.210557 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:30.221930 21994 raft_consensus.cc:482] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:30.222177 21994 raft_consensus.cc:504] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:18:30.223261 21994 leader_election.cc:288] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:18:30.226078 26425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cea6495041642f58e1eb5d26fc10df5" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:18:30.226342 26425 raft_consensus.cc:2454] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
W20240502 14:18:30.227644 26754 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:30.228031 26755 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:30.228169 26755 leader_election.cc:302] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:30.228785 21994 raft_consensus.cc:2726] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:18:30.250320 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:30.252354 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:30.272760 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:30.314145 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:30.317059 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:30.338414 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:18:30.401661 22067 raft_consensus.cc:482] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:30.401911 22067 raft_consensus.cc:504] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:18:30.402936 22067 leader_election.cc:288] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
W20240502 14:18:30.405193 26494 leader_election.cc:334] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:30.406499 26493 leader_election.cc:334] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:30.406605 26493 leader_election.cc:302] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:30.406847 22067 raft_consensus.cc:2726] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:18:30.511904 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:30.596024 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:30.598343 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:30.601891 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:30.632328 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:30.655138 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:30.663061 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:18:30.701871 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:30.702261 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:30.748872 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:18:30.754956 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:30.782749 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:30.804039 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:18:30.805137 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:30.809011 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:18:30.836026 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:30.859875 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:30.866523 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:30.897472 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:30.917989 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:18:30.922209 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:30.928817 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:30.939251 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:30.975149 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:30.988107 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:31.006506 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:31.008136 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:31.015451 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:31.076745 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:31.113129 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:31.130856 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:31.132231 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:31.151538 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:31.170915 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:31.203477 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:31.224117 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:31.229082 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:18:31.259513 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:31.286790 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:31.298911 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:31.306767 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:31.314883 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:18:31.330281 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:31.337745 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:18:31.353372 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:31.367888 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:31.385789 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:31.398857 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:31.421761 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:31.447089 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:31.461390 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:31.506235 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:31.524940 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:31.562207 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:31.577418 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:31.580260 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:31.625032 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:31.636741 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:31.704730 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:31.761636 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:31.802438 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:31.825489 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:31.922451 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:31.925040 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:31.946795 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:31.950522 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:31.951464 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:31.953914 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:31.955842 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:31.983355 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:31.995196 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:32.041144 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:32.047930 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:32.073103 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:18:32.075731 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:18:32.097369 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:32.102034 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:32.130008 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:32.147809 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:32.153075 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:32.162245 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:32.220855 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:32.238754 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:18:32.273021 22189 raft_consensus.cc:482] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:32.273221 22189 raft_consensus.cc:504] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:18:32.273991 22189 leader_election.cc:288] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:18:32.275605 26624 leader_election.cc:334] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:32.278791 26625 leader_election.cc:334] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:32.278946 26625 leader_election.cc:302] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:32.279413 22189 raft_consensus.cc:2726] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:18:32.301015 26754 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 145 similar messages]
W20240502 14:18:32.302347 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:32.319518 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:32.338712 26493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 175 similar messages]
W20240502 14:18:32.339618 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:32.351605 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:32.364634 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:32.380483 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:32.430248 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:32.461251 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:32.477550 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:32.493682 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:32.510804 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:32.518437 26362 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 95 similar messages]
W20240502 14:18:32.576825 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:32.684237 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:18:32.714740 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:32.724232 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:32.726320 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:18:32.760129 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:32.778630 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:32.809418 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:32.813128 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:32.821274 29111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 14:18:32.822839 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:18:32.863888 21465 raft_consensus.cc:482] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:32.864110 21465 raft_consensus.cc:504] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:18:32.864944 21465 leader_election.cc:288] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:18:32.866813 26493 leader_election.cc:334] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:32.867384 26494 leader_election.cc:334] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:32.867522 26494 leader_election.cc:302] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:32.867820 21465 raft_consensus.cc:2726] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:18:32.950436 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:32.974743 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:32.978981 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:32.979568 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:18:33.025696 29511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 138 similar messages]
W20240502 14:18:33.026800 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:33.044171 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:33.059249 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:33.105496 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:33.155673 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:33.181052 26624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 102 similar messages]
W20240502 14:18:33.182545 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:18:33.197643 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:33.205266 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:33.219908 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:33.249284 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:33.273260 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:18:33.281322 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:33.307060 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:33.309497 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:33.324888 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:18:33.327586 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:33.332070 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:33.413830 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:33.417670 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:33.439025 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:33.466233 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:33.467170 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:18:33.487378 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:18:33.493232 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:33.504882 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:18:33.540853 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:33.541512 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:33.552686 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:33.594251 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:33.605154 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:33.650301 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:33.667984 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:33.694798 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:33.705260 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:33.709640 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:18:33.722714 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:33.723948 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:33.725893 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:33.732714 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:33.734774 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:33.747819 22303 raft_consensus.cc:482] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:33.748256 22303 raft_consensus.cc:504] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:18:33.749090 22303 leader_election.cc:288] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:18:33.751261 26754 leader_election.cc:334] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:33.757957 26755 leader_election.cc:334] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:33.758108 26755 leader_election.cc:302] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:33.758350 22303 raft_consensus.cc:2726] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:18:33.802676 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:33.807174 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:33.837244 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:33.840014 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:18:33.859190 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:18:33.910372 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:33.951984 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:34.024942 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:34.026504 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:34.032930 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:18:34.042557 22303 raft_consensus.cc:482] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:34.042759 22303 raft_consensus.cc:504] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:18:34.043412 22303 leader_election.cc:288] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:18:34.046175 26754 leader_election.cc:334] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:34.046486 26755 leader_election.cc:334] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:34.046597 26755 leader_election.cc:302] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:34.046806 22303 raft_consensus.cc:2726] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:18:34.055723 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:34.058279 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:18:34.113742 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:34.122015 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:34.135052 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:18:34.154525 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:34.167598 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:18:34.170459 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:34.179286 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:18:34.182394 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:34.197826 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:34.221782 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:34.227787 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:34.258894 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:18:34.273157 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:34.297205 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:34.301410 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:34.301771 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:34.307631 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:34.468350 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:34.510619 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:18:34.532740 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:18:34.550712 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:18:34.583061 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:34.606685 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:18:34.684736 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:18:34.701843 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:18:34.713707 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:34.730331 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:34.733397 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:18:34.746958 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:34.756484 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:34.773844 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:18:34.788062 22656 raft_consensus.cc:482] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:34.788267 22656 raft_consensus.cc:504] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:18:34.792994 22656 leader_election.cc:288] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:18:34.793792 26815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cea6495041642f58e1eb5d26fc10df5" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:18:34.794098 26815 raft_consensus.cc:2454] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
W20240502 14:18:34.797293 26362 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:34.801549 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:34.805852 26363 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:34.806205 26363 leader_election.cc:302] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:34.806535 22656 raft_consensus.cc:2726] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:18:34.820894 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:34.845422 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:34.852710 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:18:34.855257 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:34.910988 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:34.924476 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:18:34.957792 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:34.958254 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:18:34.961040 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:34.997159 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:35.031080 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:35.049104 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:18:35.119989 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:18:35.133008 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:18:35.159474 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:35.170923 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:18:35.200421 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:35.289112 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:18:35.373279 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:18:35.453644 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:35.467108 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:35.475958 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:35.532925 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:18:35.536119 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:35.542832 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:35.587579 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:35.589761 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:35.606542 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:35.627319 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:35.707703 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:35.741560 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:18:35.763741 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:35.802171 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:18:35.825511 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:18:35.831135 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:18:35.838102 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:35.858609 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:18:35.881289 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:18:35.882514 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:18:35.882638 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:18:35.920071 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:35.944336 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:35.953562 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:35.977702 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:35.979708 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:36.017741 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:18:36.024976 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:36.033092 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:18:36.040637 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:36.047907 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:18:36.055183 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:18:36.064373 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:18:36.072925 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:18:36.074462 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:18:36.085832 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:36.103766 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:36.108878 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:36.110738 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:36.136683 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:36.149789 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:36.181594 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:36.210987 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:18:36.233451 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:18:36.293228 22657 raft_consensus.cc:482] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:36.293428 22657 raft_consensus.cc:504] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:18:36.294148 22657 leader_election.cc:288] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:18:36.295943 26494 leader_election.cc:334] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:36.296432 26493 leader_election.cc:334] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:36.296536 26493 leader_election.cc:302] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:36.326190 22657 raft_consensus.cc:2726] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:18:36.345378 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:36.399770 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:36.403514 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:36.428265 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:36.480137 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:36.508381 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:18:36.530239 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:36.537297 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:36.547420 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:18:36.588232 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:18:36.595152 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:36.629209 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:36.629595 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:36.652916 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:18:36.674983 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:18:36.675066 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:36.693871 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:18:36.698458 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:36.699872 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:36.712749 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:18:36.713976 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:36.728135 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:36.728771 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:18:36.740722 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:36.852432 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:18:36.899281 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:36.901484 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:18:36.955466 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:37.013981 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:18:37.058910 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:37.061758 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:37.115680 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:18:37.131987 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:18:37.140726 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:18:37.170492 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:37.190878 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:37.193161 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:37.212565 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:37.226011 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:18:37.229971 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:37.236460 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:37.251358 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:18:37.274521 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:37.281160 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:18:37.349575 26755 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 147 similar messages]
W20240502 14:18:37.352314 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:37.384737 26493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 173 similar messages]
W20240502 14:18:37.385600 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:18:37.428149 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:37.429714 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:18:37.486469 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:18:37.546957 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:37.550498 26362 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240502 14:18:37.578068 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:37.590662 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:37.616412 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:18:37.632809 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:37.645848 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:18:37.670487 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:37.768137 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:18:37.797510 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:18:37.812381 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:18:37.846148 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:37.847875 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:37.853173 29111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240502 14:18:37.885851 22932 raft_consensus.cc:482] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:37.886101 22932 raft_consensus.cc:504] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:18:37.887113 22932 leader_election.cc:288] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:18:37.889082 26754 leader_election.cc:334] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:37.893900 26755 leader_election.cc:334] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:37.894220 26755 leader_election.cc:302] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:37.894871 22932 raft_consensus.cc:2726] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:18:37.906890 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:18:37.966029 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:37.973594 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:37.990480 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:38.031457 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:18:38.037179 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:38.042958 29511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 138 similar messages]
W20240502 14:18:38.095265 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:18:38.104833 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:38.167271 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:38.191509 26624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240502 14:18:38.192479 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:38.216106 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:18:38.249522 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:38.253646 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:18:38.322363 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:38.325757 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:38.334646 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:38.351961 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:38.352478 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:18:38.393545 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:38.447558 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:18:38.459185 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:18:38.462836 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:18:38.470451 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:38.499401 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:18:38.508193 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:18:38.529181 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:18:38.542130 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:18:38.546922 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:38.556437 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:38.598562 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:18:38.599872 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:38.601912 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:38.612643 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:18:38.615523 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:38.629992 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:38.630326 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:18:38.642477 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:38.663193 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:38.693854 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:38.697302 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:18:38.704144 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:18:38.710014 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:18:38.721457 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:38.856551 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:38.884724 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:18:38.885789 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:38.929944 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:39.013841 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:39.025890 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:39.093904 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:39.111939 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:39.112541 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:39.134469 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:18:39.135928 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:18:39.144987 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:39.168884 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:18:39.177974 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:39.179170 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:39.186213 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:18:39.205626 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:18:39.262980 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:39.268149 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:39.272294 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:39.281750 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:18:39.286662 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:39.289130 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:18:39.308425 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:39.399739 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:18:39.400506 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:39.415242 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:39.446151 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:39.450057 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:39.484122 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:18:39.529719 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:39.545224 22932 raft_consensus.cc:482] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:39.545426 22932 raft_consensus.cc:504] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:18:39.546064 22932 leader_election.cc:288] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:18:39.549922 26754 leader_election.cc:334] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:39.550962 26755 leader_election.cc:334] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:39.551154 26755 leader_election.cc:302] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:39.551702 22932 raft_consensus.cc:2726] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:18:39.561410 21942 raft_consensus.cc:482] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:39.561662 21942 raft_consensus.cc:504] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:18:39.562345 21942 leader_election.cc:288] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:18:39.564154 26362 leader_election.cc:334] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:39.566745 26363 leader_election.cc:334] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:39.566867 26363 leader_election.cc:302] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:39.570772 23145 raft_consensus.cc:2726] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:18:39.574765 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:39.609372 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:39.610033 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:18:39.659106 23114 raft_consensus.cc:482] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:39.659292 23114 raft_consensus.cc:504] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
W20240502 14:18:39.659829 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:18:39.660079 23114 leader_election.cc:288] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:18:39.662916 26494 leader_election.cc:334] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:39.668325 26493 leader_election.cc:334] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:39.668498 26493 leader_election.cc:302] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:39.669235 23114 raft_consensus.cc:2726] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:18:39.704289 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:18:39.707826 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:18:39.712766 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:18:39.768329 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:18:39.779883 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:39.804435 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:39.839119 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:18:39.879015 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:39.879302 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:39.954825 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:18:39.961737 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:39.966712 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:39.979758 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:18:39.982455 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:39.990664 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:40.057169 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:40.126904 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:40.172950 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:18:40.244493 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:18:40.288101 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:40.306654 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:40.311388 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:40.351455 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:40.379765 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:40.400480 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:18:40.454526 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:40.457053 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:18:40.521284 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:18:40.528234 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:40.537303 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:40.568145 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:40.571256 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:18:40.573381 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:40.577947 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:40.619509 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:40.622367 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:18:40.638908 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:18:40.643265 23145 raft_consensus.cc:482] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:40.643597 23145 raft_consensus.cc:504] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:18:40.644702 23145 leader_election.cc:288] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:18:40.645747 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cea6495041642f58e1eb5d26fc10df5" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:18:40.645989 26818 raft_consensus.cc:2454] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
W20240502 14:18:40.649217 26363 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:40.659652 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:40.678833 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:18:40.679009 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:18:40.679116 26362 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:40.679229 26362 leader_election.cc:302] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:40.679517 23245 raft_consensus.cc:2726] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:18:40.707834 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:40.712352 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:40.741461 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:40.767123 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:40.780635 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:18:40.796631 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:40.931740 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:18:40.949440 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:40.950314 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:18:40.988230 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:18:41.010406 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:18:41.011132 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:41.021571 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:18:41.022559 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:18:41.030815 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:18:41.054642 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:41.058280 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:41.068056 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:18:41.072283 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:41.074936 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:41.097050 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:41.102947 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:18:41.106446 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:41.126729 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:18:41.139724 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:41.167224 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:41.168612 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:18:41.169026 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:18:41.169862 23114 raft_consensus.cc:482] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:41.170038 23114 raft_consensus.cc:504] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:18:41.170650 23114 leader_election.cc:288] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
W20240502 14:18:41.193567 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:41.193611 26493 leader_election.cc:334] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:41.195791 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:18:41.196885 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:41.204638 26494 leader_election.cc:334] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:41.204896 26494 leader_election.cc:302] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:41.205181 23114 raft_consensus.cc:2726] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:18:41.267130 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:41.507820 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:41.587180 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:41.608036 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:41.638636 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:18:41.647742 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:41.664170 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:18:41.682940 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:41.697435 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:41.732282 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:18:41.755110 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:41.761737 23370 raft_consensus.cc:482] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:41.761962 23370 raft_consensus.cc:504] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:18:41.762693 23370 leader_election.cc:288] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:18:41.764353 26624 leader_election.cc:334] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:41.776583 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:18:41.778645 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:41.779140 26625 leader_election.cc:334] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:41.779314 26625 leader_election.cc:302] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:41.779614 23370 raft_consensus.cc:2726] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:18:41.793809 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:41.795280 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:18:41.801280 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:41.820617 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:41.828853 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:41.856856 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:18:41.868366 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:18:41.891294 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:41.929749 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:18:41.942931 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:41.943068 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:41.944515 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:18:41.962743 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:41.987705 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:18:41.998569 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:42.005867 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:18:42.005965 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:42.053987 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:42.067548 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:42.068186 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:18:42.068629 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:42.159052 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:42.159322 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:18:42.195207 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:18:42.211406 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:18:42.235811 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:42.293962 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:18:42.320362 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:42.323663 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:42.334151 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:42.340723 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:42.350700 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:42.358140 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:42.372318 26755 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240502 14:18:42.397819 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:18:42.398962 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:42.416615 26493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 170 similar messages]
W20240502 14:18:42.442943 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:42.580118 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:42.632023 26363 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 94 similar messages]
W20240502 14:18:42.638427 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:18:42.708377 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:18:42.728379 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:42.772269 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:18:42.772895 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:42.773382 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:42.812705 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:42.839211 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:18:42.892776 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:42.931140 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:18:42.933200 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:42.945770 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:42.946251 29111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 81 similar messages]
I20240502 14:18:42.948848 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:42.962637 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:18:42.977164 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:42.984221 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:42.984969 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:18:43.041108 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:18:43.067552 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:18:43.070811 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:43.076782 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:43.118113 29510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 143 similar messages]
W20240502 14:18:43.132314 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:18:43.136376 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:18:43.138135 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:43.150523 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:18:43.174746 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:43.202157 26625 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 107 similar messages]
W20240502 14:18:43.209640 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:43.233824 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:43.412822 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:18:43.421259 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:18:43.424880 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:43.448750 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:43.450400 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:18:43.452142 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:18:43.454178 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:18:43.466951 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:43.489778 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:43.530407 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:18:43.536834 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:43.544349 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:43.548632 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:18:43.548964 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:18:43.555298 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:43.578483 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:18:43.581940 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:18:43.599066 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:43.600144 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:43.607115 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:43.630959 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:43.637085 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:43.644984 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:43.650102 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:18:43.665621 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:43.685474 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:18:43.733914 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:43.749451 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:18:43.818281 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:43.976323 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:43.993952 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:18:44.011449 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:44.039399 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:18:44.091145 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:44.184110 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:18:44.196056 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:18:44.214540 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:44.252080 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:44.258939 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:44.312427 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:44.320701 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:18:44.321894 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:44.328960 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:18:44.339443 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:44.339550 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:44.349777 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:44.357865 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:44.362818 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:44.371124 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:18:44.377115 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:44.428898 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:18:44.457228 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:18:44.494881 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:18:44.543759 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:18:44.562680 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:44.565411 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:44.568682 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:44.620380 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:44.662591 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:44.663892 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:44.665220 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:44.666317 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:18:44.679239 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:18:44.683416 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:44.683408 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:44.730337 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:18:44.732698 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:18:44.817471 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:18:44.846464 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:44.870904 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:44.878223 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:18:44.881973 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:44.892488 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:44.996306 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:45.115267 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:45.134014 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:45.149977 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:18:45.174315 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:45.209105 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:45.209340 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:18:45.253669 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:45.299187 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:18:45.362799 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:45.389920 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:45.416553 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:18:45.421705 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:45.423214 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:45.479884 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:45.484651 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:45.515587 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:45.516269 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:45.526021 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:45.568017 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:18:45.568938 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:18:45.582437 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:45.592669 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:45.598558 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:18:45.620693 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:18:45.649801 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:18:45.654373 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:18:45.750707 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:45.766317 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:18:45.769382 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:45.775859 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:45.781507 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:45.789710 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:18:45.796769 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:45.870774 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:45.884694 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:18:45.892824 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:45.904253 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:45.912564 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:45.984136 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:18:45.984474 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:18:45.984568 23755 raft_consensus.cc:482] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:45.984709 23755 raft_consensus.cc:504] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:18:45.985590 23755 leader_election.cc:288] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:18:45.989146 26625 leader_election.cc:334] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:45.990631 26624 leader_election.cc:334] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:45.990751 26624 leader_election.cc:302] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:45.991369 23755 raft_consensus.cc:2726] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:18:46.084465 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:18:46.093513 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:46.107105 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:18:46.111845 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:18:46.113065 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:18:46.127194 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:18:46.151672 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:46.179579 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:46.188897 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:18:46.189898 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:46.198057 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:46.214174 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:18:46.226279 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:46.226433 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:46.235337 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:18:46.252985 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:46.287309 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:18:46.335748 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:46.348603 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:46.379561 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:18:46.396121 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:46.479389 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:18:46.578855 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:18:46.696362 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:46.704864 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:46.738786 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:46.750530 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:18:46.753212 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:46.754653 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:46.766563 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:18:46.774781 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:46.806459 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:46.852802 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:46.855273 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:46.866351 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:46.882545 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:18:46.889129 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:46.921331 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:46.925546 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:46.929867 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:18:46.940861 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:46.972278 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:18:46.984444 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:47.003011 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:18:47.007181 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:18:47.011763 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:47.019215 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:47.030179 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:47.043222 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:18:47.171808 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:18:47.206524 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:18:47.213457 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:18:47.245530 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:18:47.251191 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:47.260593 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:18:47.273203 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:47.291092 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:47.336869 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:47.383280 26755 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 140 similar messages]
I20240502 14:18:47.385730 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:47.393543 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:47.416697 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:47.422437 26493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 172 similar messages]
I20240502 14:18:47.436100 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:18:47.438925 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:47.496821 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:18:47.522652 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:47.555507 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:47.641028 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:47.675897 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:47.834637 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:47.859963 26362 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 93 similar messages]
W20240502 14:18:47.867786 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:47.869617 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:18:47.901506 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:47.917475 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:47.924592 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:18:47.925828 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:47.929008 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:47.935755 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:47.935914 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:18:47.951295 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:47.951473 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:47.959004 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:47.993314 29112 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240502 14:18:47.997771 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:48.010870 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:48.018584 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:48.024796 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:18:48.049389 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:48.106421 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:48.121764 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:48.126770 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:18:48.137408 29511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240502 14:18:48.186090 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:48.186755 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:18:48.236996 26625 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240502 14:18:48.277580 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:18:48.294978 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:18:48.301940 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:18:48.312072 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:18:48.320628 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:18:48.324748 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:48.376849 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:48.386871 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:18:48.389370 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:48.402060 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:48.402073 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:18:48.409245 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:48.425837 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:48.458411 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:18:48.493988 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:48.530675 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:48.542846 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:48.594487 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:48.600559 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:48.609958 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:48.618191 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:18:48.630064 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:48.641383 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:18:48.666189 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:48.735536 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:18:48.735522 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:48.758461 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:18:48.766839 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:18:48.783875 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:18:48.795002 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:18:48.841733 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:18:48.951498 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:48.978711 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:48.982448 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:49.037825 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:18:49.038251 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:49.048185 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:18:49.075587 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:49.100646 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:49.126828 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:49.131676 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:18:49.134788 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:49.151439 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:49.152464 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:18:49.159090 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:18:49.167382 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:49.223838 23755 raft_consensus.cc:482] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:49.224041 23755 raft_consensus.cc:504] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:18:49.224777 23755 leader_election.cc:288] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:18:49.229986 26624 leader_election.cc:334] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:49.241299 26625 leader_election.cc:334] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:49.241459 26625 leader_election.cc:302] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:49.242053 23755 raft_consensus.cc:2726] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:18:49.249981 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:18:49.291224 23890 raft_consensus.cc:482] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:49.291428 23890 raft_consensus.cc:504] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:18:49.292068 23890 leader_election.cc:288] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:18:49.294318 26494 leader_election.cc:334] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:49.294550 26493 leader_election.cc:334] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:49.294631 26493 leader_election.cc:302] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:49.294886 23890 raft_consensus.cc:2726] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:18:49.322657 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:49.329540 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:18:49.360558 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:49.391614 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:49.407481 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:49.410060 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:18:49.418479 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:49.438452 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:49.448940 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:49.452878 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:49.471329 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:18:49.520798 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:49.523120 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:18:49.533116 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:18:49.576002 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:18:49.590909 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:49.663311 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:18:49.676057 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:18:49.690711 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:49.711279 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:18:49.713148 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:18:49.722924 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:18:49.803160 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:49.852838 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:49.879761 29112 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 47116.9us
I20240502 14:18:49.884567 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:49.891863 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:18:49.894593 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:49.931754 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:18:49.932085 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:49.932579 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:49.983268 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:50.020150 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:18:50.050829 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:50.085474 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:50.171785 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:50.229406 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:50.293694 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:50.356997 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:50.361856 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:50.454386 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:18:50.492506 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:18:50.503396 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:18:50.517591 24111 raft_consensus.cc:482] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:50.517788 24111 raft_consensus.cc:504] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:18:50.518792 24111 leader_election.cc:288] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:18:50.520743 26819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cea6495041642f58e1eb5d26fc10df5" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:18:50.520994 26819 raft_consensus.cc:2454] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
W20240502 14:18:50.522184 26363 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:50.524621 26362 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:50.524837 26362 leader_election.cc:302] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:50.526340 24111 raft_consensus.cc:2726] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:18:50.537918 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:50.539538 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:50.545471 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:18:50.561125 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:50.567935 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:50.569666 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:50.611264 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:18:50.612327 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:18:50.625767 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:50.628242 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:50.693420 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:18:50.696879 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:18:50.743427 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:50.747351 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:50.763732 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:18:50.770713 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:18:50.791293 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:18:50.798533 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:50.799975 24282 raft_consensus.cc:482] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:50.800206 24282 raft_consensus.cc:504] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:18:50.800336 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:50.801088 24282 leader_election.cc:288] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
W20240502 14:18:50.804374 26494 leader_election.cc:334] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:50.806740 26493 leader_election.cc:334] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:50.806947 26493 leader_election.cc:302] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:50.807526 24282 raft_consensus.cc:2726] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:18:50.822902 24111 raft_consensus.cc:482] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:50.823117 24111 raft_consensus.cc:504] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:18:50.823884 24111 leader_election.cc:288] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:18:50.834208 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:50.834718 26363 leader_election.cc:334] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:50.840799 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:50.853415 26362 leader_election.cc:334] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:50.853634 26362 leader_election.cc:302] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:50.854130 24111 raft_consensus.cc:2726] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:18:50.875536 26754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.194:36025, user_credentials={real_user=slave}} blocked reactor thread for 57062us
W20240502 14:18:50.880019 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:18:50.886173 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:50.892863 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:50.925779 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:18:50.933019 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:50.948091 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:18:50.974363 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:50.989548 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:50.998857 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:51.004297 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:51.073666 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:18:51.097445 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:51.156049 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:18:51.212955 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:18:51.301702 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:18:51.352058 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:18:51.365535 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:51.393761 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:18:51.412016 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:51.412313 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:51.416831 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:51.436941 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:51.444636 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:51.445528 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:51.453305 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:18:51.469012 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:51.472484 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:18:51.473776 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:51.477761 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:18:51.542901 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:51.554209 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:51.636216 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:18:51.667126 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:51.686755 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:51.704571 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:18:51.749344 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:51.785506 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:18:51.797168 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:18:51.821913 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:18:51.880915 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:51.892472 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:51.900099 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:51.916616 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:51.945405 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:18:51.955272 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:51.972257 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:18:51.977841 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:52.001989 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:52.023558 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:52.030508 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:18:52.035959 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:52.052649 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:18:52.064919 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:18:52.067559 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:52.080045 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:52.107991 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:18:52.142948 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:52.164012 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:52.211853 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:18:52.218689 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:18:52.259245 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:18:52.285178 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:18:52.355693 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:18:52.401472 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:52.416666 26755 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240502 14:18:52.418004 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:52.434818 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:52.464685 26494 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 173 similar messages]
W20240502 14:18:52.499045 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:52.512698 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:52.591964 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:52.644688 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:18:52.660645 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:52.699780 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:52.700989 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:18:52.705179 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:52.705204 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:52.708529 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:52.720256 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:18:52.723028 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:18:52.723140 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:52.777954 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:18:52.789762 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:52.839696 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:52.867580 26362 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240502 14:18:52.870849 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:18:53.004269 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:18:53.023386 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:53.028029 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:53.063614 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:18:53.080283 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:53.094130 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:53.123576 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:18:53.128731 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:18:53.134392 29111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 14:18:53.137678 29511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240502 14:18:53.138657 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:18:53.168581 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:18:53.168692 22707 raft_consensus.cc:482] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:53.168885 22707 raft_consensus.cc:504] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:18:53.171324 22707 leader_election.cc:288] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), 89af7f423cf44717a08d6b142f0ed49d (127.19.94.193:37985), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
W20240502 14:18:53.170696 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:53.176232 26754 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:53.176163 26427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cea6495041642f58e1eb5d26fc10df5" candidate_uuid: "e4cfb8fc5a474177af6848a6efa13a27" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "89af7f423cf44717a08d6b142f0ed49d" is_pre_election: true
I20240502 14:18:53.176470 26427 raft_consensus.cc:2454] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cfb8fc5a474177af6848a6efa13a27 in term 1.
I20240502 14:18:53.177639 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:53.181779 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:53.188968 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:18:53.194077 26755 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:53.194262 26755 leader_election.cc:302] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:53.199177 22707 raft_consensus.cc:2726] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:18:53.204825 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:18:53.200901 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:53.219935 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:18:53.220299 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:18:53.223881 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:18:53.225677 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:53.235641 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:53.244150 26625 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 105 similar messages]
I20240502 14:18:53.245122 24111 raft_consensus.cc:482] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:53.245316 24111 raft_consensus.cc:504] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:18:53.246388 24111 leader_election.cc:288] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:18:53.247644 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:53.250520 26754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.193:37985, user_credentials={real_user=slave}} blocked reactor thread for 69833.6us
W20240502 14:18:53.253252 26363 leader_election.cc:334] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:53.255334 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:18:53.276954 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:53.278236 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:53.291060 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:18:53.291185 26362 leader_election.cc:334] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:53.291287 26362 leader_election.cc:302] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:53.291723 24111 raft_consensus.cc:2726] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:18:53.429755 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:18:53.458739 24545 raft_consensus.cc:482] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:53.458971 24545 raft_consensus.cc:504] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:18:53.459818 24545 leader_election.cc:288] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:18:53.461831 26755 leader_election.cc:334] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:53.462203 26754 leader_election.cc:334] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:53.462328 26754 leader_election.cc:302] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:53.463021 24545 raft_consensus.cc:2726] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:18:53.535873 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:53.576072 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:53.593294 24545 raft_consensus.cc:482] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:53.593472 24545 raft_consensus.cc:504] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:18:53.594233 24545 leader_election.cc:288] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:18:53.596127 26754 leader_election.cc:334] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:53.602596 26755 leader_election.cc:334] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:53.602763 26755 leader_election.cc:302] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:53.603456 24545 raft_consensus.cc:2726] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:18:53.624786 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:53.639431 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:53.667598 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:18:53.683197 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:18:53.726714 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:18:53.756608 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:53.783556 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:18:53.787285 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:53.788887 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:53.796995 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:18:53.813710 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:53.814567 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:53.828099 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:18:53.832670 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:53.846598 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:18:53.886756 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:53.890931 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:53.973778 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:54.042461 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:18:54.079439 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:54.109714 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:18:54.129316 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:18:54.139281 24282 raft_consensus.cc:482] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:54.139501 24282 raft_consensus.cc:504] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:18:54.140221 24282 leader_election.cc:288] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
W20240502 14:18:54.147050 26494 leader_election.cc:334] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:54.151599 26493 leader_election.cc:334] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:54.151791 26493 leader_election.cc:302] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:54.152176 24282 raft_consensus.cc:2726] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:18:54.207885 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:54.208947 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:54.214251 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:54.236992 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:54.248379 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:54.257834 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:18:54.267867 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:18:54.284814 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:18:54.291599 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:54.292059 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:18:54.308557 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:54.321561 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:18:54.326282 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:54.343283 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:18:54.370358 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:54.380276 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:54.391587 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:18:54.395769 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:18:54.398507 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:54.416046 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:18:54.473060 24468 raft_consensus.cc:482] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
W20240502 14:18:54.473315 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:54.473265 24468 raft_consensus.cc:504] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:18:54.474023 24468 leader_election.cc:288] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:18:54.475440 26493 leader_election.cc:334] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:54.477586 26494 leader_election.cc:334] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:54.477739 26494 leader_election.cc:302] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:54.478206 24468 raft_consensus.cc:2726] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:18:54.518461 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:54.532899 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:18:54.579320 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:18:54.593376 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:18:54.616930 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:18:54.616900 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:18:54.619323 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:54.632277 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:54.694077 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:54.719055 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:18:54.741894 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:54.769061 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:54.903287 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:18:54.944020 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:54.954758 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:54.980887 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:55.011988 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:55.019438 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:18:55.023053 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:55.030500 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:55.045677 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:18:55.046002 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:55.077584 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:55.094548 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:18:55.103365 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:55.250496 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:55.256089 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:55.276826 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:18:55.288975 24282 raft_consensus.cc:482] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:55.289167 24282 raft_consensus.cc:504] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:18:55.289804 24282 leader_election.cc:288] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:18:55.291944 26493 leader_election.cc:334] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:55.302019 26494 leader_election.cc:334] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:55.302173 26494 leader_election.cc:302] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:55.302414 24282 raft_consensus.cc:2726] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:18:55.333870 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:18:55.341619 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:55.403468 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:18:55.407419 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:55.408315 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:55.423872 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:55.448303 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:18:55.454975 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:18:55.474086 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:55.510352 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:18:55.527663 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:55.527988 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:18:55.534322 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:18:55.539573 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:55.579932 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:18:55.613008 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:55.618516 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:18:55.625043 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:55.631140 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:55.637486 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:55.643575 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:18:55.647572 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:55.697669 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:55.711982 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:55.763464 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:18:55.865043 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:18:55.865034 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:18:55.869573 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:18:55.871985 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:18:55.879034 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:18:55.884497 24545 raft_consensus.cc:482] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:55.884851 24545 raft_consensus.cc:504] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:18:55.885679 24545 leader_election.cc:288] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:18:55.888681 26755 leader_election.cc:334] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:55.888901 26754 leader_election.cc:334] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:55.889072 26754 leader_election.cc:302] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:55.889665 24545 raft_consensus.cc:2726] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:18:55.971237 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:55.972117 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:18:56.016274 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:56.093317 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:56.095455 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:56.111228 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:18:56.116611 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:56.117836 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:56.158574 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:18:56.165180 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:56.167263 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:18:56.167759 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:56.168231 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:18:56.209918 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:56.224207 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:56.240612 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:18:56.346284 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:18:56.391903 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:18:56.429450 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:18:56.431620 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:56.490473 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:56.497792 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:56.560668 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:56.575783 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:18:56.580063 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:18:56.592643 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:18:56.610392 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:56.633682 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:56.641562 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:18:56.719517 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:56.754897 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:18:56.771345 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:56.786845 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:56.793402 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:18:56.836481 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:56.852757 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:18:56.853241 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:56.869436 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:18:56.881942 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:56.887212 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:56.888686 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:18:56.904269 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:56.945376 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:56.951409 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:18:56.970769 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:56.996698 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:18:56.996570 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:18:56.998528 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:18:57.015569 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:18:57.021059 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:57.035275 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:57.131062 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:57.157956 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:18:57.213004 24918 raft_consensus.cc:482] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:57.213220 24918 raft_consensus.cc:504] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:18:57.213901 24918 leader_election.cc:288] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:18:57.215596 26625 leader_election.cc:334] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:57.218855 26624 leader_election.cc:334] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:57.219056 26624 leader_election.cc:302] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:57.220450 24918 raft_consensus.cc:2726] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:18:57.248942 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:18:57.278596 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:18:57.321394 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:57.360854 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:57.376511 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:57.380194 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:57.400735 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:57.438457 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:18:57.460008 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:18:57.499878 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:57.513504 26755 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 151 similar messages]
W20240502 14:18:57.515183 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:57.530807 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:57.555253 26493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 180 similar messages]
I20240502 14:18:57.583062 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:57.583534 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:57.585276 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:18:57.635200 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:57.638808 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:57.649807 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:18:57.688371 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:57.777873 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:57.781649 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:57.799844 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:18:57.807096 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:57.817559 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:18:57.935487 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:18:57.937466 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:57.938380 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:57.946542 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:18:57.950659 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:57.996894 26363 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240502 14:18:57.998178 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:58.066468 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:18:58.071069 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:58.076928 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:58.081281 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:58.085093 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:58.089362 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:18:58.090914 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:18:58.100905 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:58.125314 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:58.140830 29511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 139 similar messages]
I20240502 14:18:58.154860 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:58.162938 29112 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 84 similar messages]
W20240502 14:18:58.172858 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:18:58.193908 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:18:58.199168 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:18:58.227540 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:58.244421 26625 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240502 14:18:58.257967 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:18:58.258612 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:58.331298 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:18:58.338263 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:18:58.372786 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:58.387485 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:18:58.389981 25039 raft_consensus.cc:482] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:18:58.390139 25039 raft_consensus.cc:504] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:18:58.390719 25039 leader_election.cc:288] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
W20240502 14:18:58.393353 26494 leader_election.cc:334] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:58.393836 26493 leader_election.cc:334] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:18:58.393922 26493 leader_election.cc:302] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:18:58.394414 25039 raft_consensus.cc:2726] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:18:58.398983 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:18:58.423830 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:58.480993 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:18:58.547564 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:58.549436 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:58.561259 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:18:58.593181 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:18:58.661439 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:58.692250 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:18:58.706266 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:18:58.728673 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:58.754772 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:58.760462 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:58.761483 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:58.765167 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:18:58.798789 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:18:58.808552 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:58.822263 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:58.893452 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:18:58.928010 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:18:58.953270 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:18:58.969681 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:59.009595 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:59.033957 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:59.053130 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:18:59.089815 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:59.119640 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:18:59.185557 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:18:59.240470 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:59.270356 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:18:59.273887 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:18:59.283407 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:18:59.288693 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:59.289238 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:18:59.290665 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:59.303380 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:59.321558 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:59.321856 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:18:59.343222 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:59.349282 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:18:59.384276 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:18:59.386178 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:18:59.387321 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:18:59.419226 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:18:59.454453 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:18:59.493216 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:18:59.509186 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:18:59.511409 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:18:59.524016 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:18:59.573184 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:18:59.575855 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:18:59.588155 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:59.659548 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:18:59.660507 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:18:59.685197 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:18:59.686496 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:18:59.816098 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:18:59.855559 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:18:59.863631 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:18:59.886471 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:18:59.907001 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:18:59.908870 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:18:59.915632 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:18:59.918676 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:18:59.927142 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:18:59.957059 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:18:59.975131 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:18:59.975272 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:18:59.988050 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:19:00.011394 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:19:00.172531 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:19:00.186255 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:19:00.187988 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:19:00.207427 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:19:00.250485 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:00.261145 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:19:00.287601 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:19:00.349928 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:19:00.372257 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:19:00.386523 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:19:00.386966 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:00.411839 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:19:00.547104 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:19:00.557924 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:19:00.605122 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:19:00.620579 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:19:00.658653 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:19:00.678704 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:19:00.690770 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:19:00.698397 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:00.701153 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:19:00.718870 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:19:00.719964 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:19:00.723232 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:00.745959 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:19:00.755060 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:00.761926 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:19:00.769042 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:19:00.770560 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:19:00.786381 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:19:00.807195 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:19:00.825881 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:19:00.852249 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:19:00.868134 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:19:00.911600 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:19:00.935443 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:00.946871 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:19:01.066746 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:19:01.092866 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:19:01.100283 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:19:01.101603 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:19:01.127252 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:19:01.180574 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:19:01.214718 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:19:01.264606 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:19:01.264740 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:01.280761 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:19:01.297398 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:19:01.323715 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:19:01.329031 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:19:01.338806 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:19:01.360971 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:19:01.394765 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:19:01.412249 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:19:01.512298 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:19:01.545818 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:19:01.560194 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:01.639487 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:19:01.650637 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:19:01.669663 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:19:01.693475 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:19:01.788094 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:19:01.797219 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:19:01.812494 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:19:01.815166 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:19:01.832367 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:01.846060 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:19:01.854255 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:19:01.855306 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:19:01.875156 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:19:01.875802 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:19:01.881042 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:19:01.894176 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:19:01.911742 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:19:01.917732 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:19:01.920450 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:19:01.927547 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:19:01.935909 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:19:01.938025 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:19:01.990917 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:19:02.049294 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:19:02.053403 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:19:02.068313 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:19:02.140435 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:19:02.169121 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:02.174309 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:19:02.189595 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:19:02.238867 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:02.255978 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:19:02.307562 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:19:02.328522 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:19:02.380900 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:19:02.400686 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:19:02.546255 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:19:02.547777 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:19:02.565382 26493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 169 similar messages]
W20240502 14:19:02.593933 26754 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 135 similar messages]
W20240502 14:19:02.605170 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:02.608866 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:19:02.617352 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:19:02.627535 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:19:02.637521 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:02.656553 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:19:02.659739 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:19:02.662781 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:19:02.687353 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:19:02.716907 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:19:02.721773 25443 raft_consensus.cc:482] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:19:02.721982 25443 raft_consensus.cc:504] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:19:02.722645 25443 leader_election.cc:288] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:19:02.724342 26493 leader_election.cc:334] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:02.733439 26494 leader_election.cc:334] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:19:02.733580 26494 leader_election.cc:302] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:19:02.733932 25443 raft_consensus.cc:2726] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:19:02.737232 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:19:02.748427 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:19:02.803669 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:19:02.820468 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:19:02.867143 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:19:02.873355 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:19:02.919307 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:19:02.953980 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:19:02.981747 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:19:03.039250 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:19:03.088446 26362 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 14:19:03.089854 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:19:03.115306 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:19:03.118680 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:19:03.129905 25445 raft_consensus.cc:482] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:19:03.131552 25445 raft_consensus.cc:504] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:19:03.132444 25445 leader_election.cc:288] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
W20240502 14:19:03.134454 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:19:03.136298 26362 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:19:03.137529 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cea6495041642f58e1eb5d26fc10df5" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:19:03.137812 26818 raft_consensus.cc:2454] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
W20240502 14:19:03.138830 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:03.141556 26363 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:19:03.141719 26363 leader_election.cc:302] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:19:03.142140 25445 raft_consensus.cc:2726] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:19:03.160748 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:19:03.181648 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:19:03.189522 29111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 14:19:03.189661 29511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 135 similar messages]
W20240502 14:19:03.191710 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:19:03.224856 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:19:03.229517 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:19:03.300196 26625 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 98 similar messages]
I20240502 14:19:03.303876 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:19:03.316224 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:19:03.319115 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:19:03.321882 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:19:03.330183 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:19:03.343755 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:19:03.360021 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:19:03.370748 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:19:03.374722 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:19:03.399901 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:19:03.410653 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:19:03.440896 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:19:03.481722 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:19:03.482453 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:19:03.529378 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:03.542634 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:19:03.582669 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:03.644220 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:19:03.647691 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:19:03.654616 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:19:03.670586 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:19:03.734195 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:19:03.753285 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:19:03.794348 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:19:03.811542 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:19:03.822628 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:19:03.825565 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:19:03.839288 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:19:03.853315 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:03.884812 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:19:03.911754 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:19:03.922233 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:19:03.973548 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:19:03.989995 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:19:04.049693 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:19:04.053454 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:19:04.097379 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:19:04.103463 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:04.103545 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:19:04.109493 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:04.116626 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:19:04.139086 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:19:04.257625 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:19:04.274953 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:19:04.329684 25632 raft_consensus.cc:482] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:19:04.329886 25632 raft_consensus.cc:504] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:19:04.330615 25632 leader_election.cc:288] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:19:04.337106 26754 leader_election.cc:334] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:04.354315 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:19:04.379338 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:19:04.382730 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:19:04.418653 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:19:04.434643 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:19:04.461601 29510 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.197:38715, user_credentials={real_user=slave}} blocked reactor thread for 49974.6us
W20240502 14:19:04.522015 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:19:04.555184 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:19:04.567319 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:19:04.594081 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:19:04.602013 26755 leader_election.cc:334] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:19:04.604380 26755 leader_election.cc:302] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:19:04.605250 25632 raft_consensus.cc:2726] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:19:04.605513 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:19:04.607031 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:19:04.611958 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:19:04.612497 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:19:04.613092 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:04.617142 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:19:04.626875 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:04.646153 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:19:04.646582 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:19:04.706516 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:19:04.706954 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:04.722020 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:19:04.744891 29511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.199:41781, user_credentials={real_user=slave}} blocked reactor thread for 150813us
W20240502 14:19:04.747727 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:04.765472 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:19:04.787954 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:19:04.792515 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:19:04.814455 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:19:04.900022 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:19:05.014669 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:05.058547 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:19:05.107319 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:19:05.143270 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:19:05.180413 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:19:05.180956 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:19:05.181967 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:05.187922 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:05.192512 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:19:05.209295 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:19:05.222611 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:19:05.240600 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:19:05.263154 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:19:05.283805 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:19:05.331804 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:19:05.332540 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:19:05.354288 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:19:05.381945 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:19:05.390664 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:19:05.414063 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:19:05.427973 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:19:05.462800 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:19:05.539227 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:19:05.553583 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:19:05.575042 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:05.585873 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:19:05.601691 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:19:05.614625 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:05.656491 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:19:05.661685 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:19:05.681818 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:19:05.686491 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:19:05.693677 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:05.696143 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:19:05.750330 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:19:05.751542 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:19:05.794432 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:19:05.861629 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:19:05.866670 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:19:05.866910 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:19:05.876706 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:19:05.891556 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:19:05.919296 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:19:05.949241 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:19:05.950769 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:19:06.011683 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:19:06.054682 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:06.069301 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:19:06.083932 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:19:06.104148 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:19:06.140229 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:19:06.158197 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:19:06.168422 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:19:06.195719 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:19:06.216584 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:06.219203 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:19:06.290365 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:19:06.339322 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:06.343017 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:19:06.356256 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:19:06.356657 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:19:06.362128 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:19:06.363591 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:19:06.365803 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:19:06.377828 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:19:06.395392 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:19:06.398087 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:19:06.400738 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:19:06.487913 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:19:06.584959 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:19:06.614014 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:06.614565 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:06.626852 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:19:06.627380 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:19:06.729344 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:19:06.752285 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:19:06.821719 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:19:06.869745 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:19:06.905095 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:19:06.919081 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:19:06.935942 25894 raft_consensus.cc:482] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:19:06.936220 25894 raft_consensus.cc:504] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "69ab78b2a1b74d2bbf2cdeafa0a4af23" member_type: VOTER last_known_addr { host: "127.19.94.195" port: 44325 } }
I20240502 14:19:06.937477 25894 leader_election.cc:288] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:19:06.940838 26624 leader_election.cc:334] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:06.941063 26625 leader_election.cc:334] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:19:06.941164 26625 leader_election.cc:302] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69ab78b2a1b74d2bbf2cdeafa0a4af23; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:19:06.941406 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:06.941956 25684 raft_consensus.cc:2726] T e93acdb9427943938f4c12ae2bf2c594 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:19:06.944262 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:19:06.946271 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:19:06.963678 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:19:06.986649 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:19:07.001346 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:19:07.006455 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:19:07.019923 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:19:07.022256 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:07.029654 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:19:07.046563 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:19:07.064801 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:19:07.099515 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:07.146731 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:19:07.147315 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:19:07.150925 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:19:07.210150 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:19:07.233830 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:07.247196 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:19:07.257608 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:19:07.296909 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:19:07.308154 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:19:07.449321 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:19:07.451361 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:19:07.455879 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:19:07.501892 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:19:07.526979 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:19:07.543422 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:19:07.554396 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:19:07.559741 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:19:07.569087 26493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 172 similar messages]
I20240502 14:19:07.572578 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:19:07.573773 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:19:07.617322 26754 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240502 14:19:07.618402 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:19:07.628818 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:07.657022 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:07.734958 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:19:07.753540 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:19:07.877235 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:19:07.946125 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:19:07.953388 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:19:07.968509 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:19:07.973126 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:19:07.982161 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:19:08.016161 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:19:08.016947 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:19:08.020798 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:19:08.047096 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:19:08.054256 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:19:08.056278 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:08.062644 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:19:08.072825 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:19:08.089855 26362 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 92 similar messages]
I20240502 14:19:08.100212 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:19:08.100502 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:08.110694 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:08.112569 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:19:08.113950 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:19:08.141937 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:19:08.145270 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:08.147783 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:19:08.160189 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:19:08.195505 29511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 138 similar messages]
W20240502 14:19:08.196596 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:19:08.215489 29112 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 14:19:08.261953 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:19:08.293174 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:19:08.326617 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:19:08.337927 26625 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240502 14:19:08.398486 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:19:08.410872 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:19:08.436461 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:19:08.451735 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:19:08.457695 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:19:08.516552 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:19:08.516718 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:19:08.544092 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:19:08.545442 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:08.553694 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:19:08.564154 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:19:08.570487 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:19:08.596689 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:19:08.635972 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:19:08.686442 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:19:08.687943 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:19:08.700452 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:19:08.713973 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:19:08.728827 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:19:08.742664 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:19:08.747260 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:19:08.752688 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:19:08.755695 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:19:08.781808 26363 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.200:46097, user_credentials={real_user=slave}} blocked reactor thread for 55698.6us
W20240502 14:19:08.839200 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:19:08.840906 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:19:08.847663 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:19:08.853914 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:19:08.882128 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:19:09.022142 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:19:09.052695 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:09.053186 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:19:09.060636 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:09.063516 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:19:09.113265 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:19:09.194880 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:19:09.259055 26006 raft_consensus.cc:482] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:19:09.259371 26006 raft_consensus.cc:504] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:19:09.260442 26006 leader_election.cc:288] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:19:09.265367 26494 leader_election.cc:334] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:09.266963 26493 leader_election.cc:334] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:19:09.267124 26493 leader_election.cc:302] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:19:09.267896 26056 raft_consensus.cc:2726] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:19:09.269737 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:19:09.301589 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:19:09.323025 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:19:09.391645 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:19:09.397840 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:19:09.435066 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:09.445436 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:09.468076 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:19:09.481026 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:19:09.564246 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:09.570339 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:19:09.573930 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:19:09.591632 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:19:09.594544 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:09.609398 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:19:09.621110 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:19:09.626053 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:19:09.628345 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:19:09.628710 26006 raft_consensus.cc:482] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:19:09.628917 26006 raft_consensus.cc:504] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:19:09.629585 26006 leader_election.cc:288] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:19:09.642799 26494 leader_election.cc:334] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:09.647316 26493 leader_election.cc:334] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:19:09.647526 26493 leader_election.cc:302] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:19:09.648663 26006 raft_consensus.cc:2726] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:19:09.658792 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:19:09.670621 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:19:09.705471 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:19:09.720234 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:19:09.739068 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:19:09.777611 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:19:09.806280 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:19:09.810954 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:19:09.857218 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:19:09.897101 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:09.897570 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:09.921844 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:19:09.987834 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:19:10.001327 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:19:10.017637 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:19:10.106842 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:19:10.133164 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:19:10.170431 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:10.194175 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:19:10.199443 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:19:10.204898 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:10.212342 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:19:10.236621 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:19:10.264626 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:19:10.354713 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:10.455049 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:10.475962 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:19:10.486505 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:19:10.526599 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:19:10.529510 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:19:10.542070 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:19:10.556092 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:19:10.561748 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:19:10.562785 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:19:10.563671 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:19:10.565586 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:19:10.593538 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:19:10.627982 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:19:10.660588 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:19:10.672639 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:19:10.718362 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:19:10.749361 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:19:10.765518 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:19:10.778772 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:19:10.783954 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:19:10.795325 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:19:10.797768 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:19:10.801232 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:19:10.812530 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:19:10.820484 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:19:10.831964 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:19:10.874732 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:19:10.875005 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:19:10.879077 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:19:10.889465 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:19:10.896054 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:19:10.921371 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:19:10.927614 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:19:10.945024 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:10.948516 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:19:10.949851 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:19:10.957626 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:19:11.001732 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:19:11.014071 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:19:11.060899 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:19:11.087759 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:19:11.263396 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:19:11.280110 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:19:11.280179 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:11.295698 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:19:11.385778 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:11.386458 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:19:11.391269 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:19:11.394816 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:19:11.422797 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:19:11.441833 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:11.456125 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:19:11.472112 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:19:11.536870 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:19:11.555416 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:19:11.558665 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:19:11.560873 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:19:11.563071 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:11.578513 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:19:11.585963 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:19:11.592741 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:19:11.619199 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:19:11.744750 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:19:11.771215 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:19:11.771759 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:19:11.784943 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:19:11.866405 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:11.966589 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:19:12.003556 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:19:12.026453 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:19:12.057504 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:12.058660 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:19:12.095669 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:19:12.101554 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:19:12.115393 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:19:12.126878 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:19:12.131354 25990 raft_consensus.cc:482] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:19:12.131567 25990 raft_consensus.cc:504] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:19:12.132323 25990 leader_election.cc:288] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:19:12.135162 26755 leader_election.cc:334] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:12.167721 26754 leader_election.cc:334] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:19:12.168028 26754 leader_election.cc:302] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:19:12.168359 25990 raft_consensus.cc:2726] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:19:12.169092 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:19:12.180722 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:19:12.180990 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:19:12.194257 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:19:12.198931 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:19:12.202885 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:19:12.210326 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:19:12.210695 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:19:12.257032 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:19:12.333731 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:19:12.360324 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:19:12.403597 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:19:12.447463 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:19:12.497229 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:12.558447 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:12.563271 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:19:12.564043 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:19:12.572371 26493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 170 similar messages]
W20240502 14:19:12.621870 26754 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 144 similar messages]
W20240502 14:19:12.642053 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:19:12.674829 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:19:12.722448 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:19:12.774488 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:19:12.805878 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:19:12.838807 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:12.848301 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:19:12.875721 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:19:12.881621 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:19:12.895000 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:19:12.911113 25990 raft_consensus.cc:482] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:19:12.911317 25990 raft_consensus.cc:504] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:19:12.912024 25990 leader_election.cc:288] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:19:12.914677 26754 leader_election.cc:334] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:12.914903 26755 leader_election.cc:334] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:19:12.915011 26755 leader_election.cc:302] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:19:12.915422 25990 raft_consensus.cc:2726] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:19:12.924342 26968 raft_consensus.cc:482] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:19:12.924562 26968 raft_consensus.cc:504] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:19:12.925312 26968 leader_election.cc:288] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:19:12.928105 26362 leader_election.cc:334] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:12.940701 26363 leader_election.cc:334] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:19:12.940868 26363 leader_election.cc:302] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:19:12.941094 26968 raft_consensus.cc:2726] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:19:12.972908 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:19:12.983397 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:19:12.983661 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:12.994979 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:19:13.008443 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:19:13.017328 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:19:13.022172 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:19:13.030545 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:19:13.049975 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:19:13.059638 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:19:13.062438 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:19:13.065654 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:13.073163 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:19:13.085181 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:19:13.109436 26362 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 14:19:13.113507 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:19:13.117779 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:19:13.125767 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:19:13.231698 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:19:13.263850 29511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240502 14:19:13.299544 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:19:13.317152 26974 raft_consensus.cc:482] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:19:13.317353 26974 raft_consensus.cc:504] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:19:13.318089 26974 leader_election.cc:288] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:19:13.321874 26493 leader_election.cc:334] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:13.324980 26494 leader_election.cc:334] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:19:13.325146 26494 leader_election.cc:302] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:19:13.325776 26006 raft_consensus.cc:2726] T a511c4f2b4c14f4a90872bf0093c035c P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:19:13.326705 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:19:13.331710 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:19:13.339844 26624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240502 14:19:13.341058 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:19:13.344758 29112 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 14:19:13.421329 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:19:13.447371 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:19:13.456764 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:19:13.457661 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:19:13.463697 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:19:13.481539 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:19:13.495813 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:19:13.499070 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:19:13.522390 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:13.554142 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:19:13.554538 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:19:13.557313 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:19:13.564826 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:13.570468 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:19:13.586807 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:19:13.597227 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:19:13.598722 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:19:13.614012 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:19:13.642458 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:13.654206 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:19:13.716848 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:19:13.726490 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:19:13.747046 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:19:13.957392 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:19:13.962801 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:19:13.971797 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:19:13.986388 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:14.002673 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:19:14.029610 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:19:14.042506 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:19:14.091354 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:19:14.116791 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:19:14.152146 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:19:14.153671 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:19:14.157934 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:14.158792 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:19:14.161528 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:19:14.167630 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:14.187764 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:19:14.234126 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:19:14.237103 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:19:14.238725 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:19:14.293267 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:19:14.316557 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:19:14.321415 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:19:14.391139 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:19:14.399396 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:19:14.441257 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:19:14.460000 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:19:14.562376 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:14.583521 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:19:14.597254 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:19:14.612665 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:19:14.633927 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:14.658262 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:19:14.674048 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:19:14.690945 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:19:14.731069 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:19:14.748844 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:14.775085 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:19:14.776561 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:19:14.776882 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:14.779947 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:19:14.864832 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:19:14.877282 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:19:14.899823 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:19:14.900061 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:19:14.911432 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:19:14.918802 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:19:14.951335 25635 raft_consensus.cc:482] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:19:14.951557 25635 raft_consensus.cc:504] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:19:14.952662 25635 leader_election.cc:288] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), e4cfb8fc5a474177af6848a6efa13a27 (127.19.94.196:32843), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
I20240502 14:19:14.953799 26819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cea6495041642f58e1eb5d26fc10df5" candidate_uuid: "89af7f423cf44717a08d6b142f0ed49d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4cfb8fc5a474177af6848a6efa13a27" is_pre_election: true
I20240502 14:19:14.954195 26819 raft_consensus.cc:2454] T 4cea6495041642f58e1eb5d26fc10df5 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af7f423cf44717a08d6b142f0ed49d in term 1.
W20240502 14:19:14.955638 26362 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:14.956162 26363 leader_election.cc:334] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:19:14.956306 26363 leader_election.cc:302] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d, e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:19:14.956704 25635 raft_consensus.cc:2726] T 4cea6495041642f58e1eb5d26fc10df5 P 89af7f423cf44717a08d6b142f0ed49d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:19:14.961022 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:19:15.022516 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:15.023213 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:19:15.062000 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:19:15.065680 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:15.188055 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:19:15.264047 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:15.269418 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:19:15.305415 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:19:15.337626 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:19:15.348026 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:19:15.377873 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:19:15.380434 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:19:15.382714 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:19:15.387307 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:19:15.439985 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:19:15.482707 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:19:15.497512 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:15.528748 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:19:15.564138 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:19:15.583693 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:15.587272 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:19:15.606760 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:19:15.608055 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:19:15.612002 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:19:15.634133 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:19:15.646173 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:19:15.656838 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:19:15.668176 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:19:15.684984 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:19:15.697607 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:19:15.705801 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:19:15.743242 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:19:15.829218 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:19:15.833254 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:19:15.843138 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:19:15.897560 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:19:15.944835 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:19:15.950932 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:19:15.988730 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:19:16.020166 29511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.199:41781, user_credentials={real_user=slave}} blocked reactor thread for 75376.5us
W20240502 14:19:16.025334 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:19:16.054428 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:19:16.055120 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:19:16.058692 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:19:16.060688 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:19:16.085136 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:19:16.104789 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:19:16.116894 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:19:16.117149 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:16.132481 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:16.133795 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:19:16.144438 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:19:16.159122 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:19:16.174671 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:19:16.195130 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:19:16.197904 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:19:16.275532 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:19:16.278861 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:19:16.297436 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 14:19:16.304306 26006 raft_consensus.cc:482] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:19:16.304515 26006 raft_consensus.cc:504] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
I20240502 14:19:16.305217 26006 leader_election.cc:288] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
W20240502 14:19:16.307020 26494 leader_election.cc:334] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:16.311970 26493 leader_election.cc:334] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:19:16.312106 26493 leader_election.cc:302] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:19:16.312431 26006 raft_consensus.cc:2726] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:19:16.359871 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:19:16.483685 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:19:16.506522 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:19:16.589309 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:16.648356 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:19:16.684303 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:16.704473 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:19:16.705718 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:19:16.726341 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:19:16.729939 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:19:16.733896 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 14:19:16.741314 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:16.753408 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:16.759279 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:19:16.795073 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:19:16.797020 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:19:16.828869 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:19:16.829350 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:19:16.850857 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:19:16.877552 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:19:16.905159 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:19:16.915853 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:19:16.928681 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:19:16.968509 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:19:16.982513 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:19:16.989286 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:19:17.036137 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:19:17.099821 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:17.126041 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:19:17.133373 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:19:17.186048 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:19:17.189355 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:19:17.202210 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:19:17.225003 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:19:17.227574 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 14:19:17.240664 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:19:17.240948 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:19:17.246834 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:19:17.265120 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:19:17.286592 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:17.318190 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:19:17.340440 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:19:17.352279 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:19:17.368988 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:19:17.371670 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:19:17.412487 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:19:17.458101 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:19:17.576267 26493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 175 similar messages]
W20240502 14:19:17.634760 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:19:17.663295 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:19:17.695421 26754 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240502 14:19:17.712849 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:17.785790 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:19:17.793641 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:17.816668 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:19:17.861338 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:19:17.871024 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:19:17.886039 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:19:17.919288 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:19:17.919850 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:19:17.923115 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:19:17.927862 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:17.944679 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:19:17.984709 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:19:17.988474 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:19:17.988492 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:19:18.081763 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:19:18.094666 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:19:18.110044 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:19:18.113464 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:19:18.159291 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:19:18.168351 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:19:18.178598 26363 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 14:19:18.179872 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:19:18.256472 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:19:18.265445 29511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240502 14:19:18.268441 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:19:18.273938 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:18.287582 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:19:18.306238 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:19:18.317935 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:19:18.323596 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:19:18.351449 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:19:18.353843 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:18.358706 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:19:18.367264 26625 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240502 14:19:18.373015 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:19:18.380434 29111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 14:19:18.423214 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:19:18.436954 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:19:18.471239 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:18.523067 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:19:18.524286 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:19:18.527266 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:19:18.551409 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:19:18.578979 26754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.197:38715, user_credentials={real_user=slave}} blocked reactor thread for 53820.1us
I20240502 14:19:18.603286 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:19:18.607309 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:19:18.623149 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:19:18.625808 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:19:18.648480 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:19:18.656599 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:19:18.667061 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:19:18.667932 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:19:18.733390 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:19:18.740700 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:19:18.815254 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:18.825899 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:19:18.847165 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:18.935482 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:19:18.974881 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:19:18.990710 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:19:18.994726 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:19:18.999315 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:19:19.006554 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 14:19:19.017608 27617 raft_consensus.cc:482] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:19:19.017879 27617 raft_consensus.cc:504] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "89af7f423cf44717a08d6b142f0ed49d" member_type: VOTER last_known_addr { host: "127.19.94.193" port: 37985 } }
I20240502 14:19:19.018831 27617 leader_election.cc:288] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:19:19.025120 26362 leader_election.cc:334] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:19:19.069558 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:19:19.075229 26363 leader_election.cc:334] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:19:19.075634 26363 leader_election.cc:302] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89af7f423cf44717a08d6b142f0ed49d; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:19:19.075996 27617 raft_consensus.cc:2726] T cf808a060536445495a70de638438f87 P 89af7f423cf44717a08d6b142f0ed49d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:19:19.111512 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:19:19.111799 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:19.130012 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:19.131606 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:19:19.171914 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:19:19.174052 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:19:19.179085 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:19:19.187945 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:19:19.222357 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:19:19.252214 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:19:19.295208 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:19:19.308715 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:19.334657 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:19.370450 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:19:19.371928 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:19:19.373593 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:19:19.388437 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:19:19.446460 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:19:19.460494 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:19:19.478813 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:19:19.507537 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:19:19.556795 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:19:19.584997 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:19:19.631610 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:19:19.669322 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:19:19.682354 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:19:19.713755 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:19:19.725643 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:19:19.741595 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:19:19.748631 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:19:19.762951 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:19.780164 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:19:19.813776 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:19:19.828969 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:19:19.844935 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:19.865325 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:19:19.870314 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:19:19.953758 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:19:19.978346 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:19:19.983847 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:19:19.988957 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:19:20.005910 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:19:20.068632 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:19:20.109920 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:19:20.178416 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:19:20.182605 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:20.202164 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:19:20.251621 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:19:20.273739 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:19:20.373811 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:19:20.377723 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:20.397060 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:19:20.411733 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:19:20.430996 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:19:20.470736 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:19:20.501325 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:19:20.516136 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:19:20.527760 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:19:20.544345 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:19:20.547258 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:19:20.554997 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:19:20.558838 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:19:20.565727 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:19:20.594805 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:19:20.658605 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:19:20.693379 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:19:20.712633 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:19:20.712779 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:19:20.723882 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:19:20.731748 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:19:20.735196 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:19:20.760057 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:20.760200 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:19:20.799925 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:19:20.914461 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:19:20.936040 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:19:20.953924 27916 raft_consensus.cc:482] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:19:20.954303 27916 raft_consensus.cc:504] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:19:20.955284 27916 leader_election.cc:288] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:19:20.960222 26755 leader_election.cc:334] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:20.963045 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:19:20.963292 26754 leader_election.cc:334] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:19:20.963413 26754 leader_election.cc:302] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:19:20.963886 27916 raft_consensus.cc:2726] T 63b0005e85cc4c4fab7ba8f2c728ffc2 P e4cfb8fc5a474177af6848a6efa13a27 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:19:20.984498 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:19:20.996497 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:19:21.002049 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:19:21.076558 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:19:21.077996 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:19:21.078450 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:21.090188 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:19:21.100284 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:19:21.125865 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:19:21.138849 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:19:21.143281 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:19:21.147298 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:19:21.172998 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:21.175335 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:19:21.229100 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:19:21.266018 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:19:21.309019 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:19:21.387187 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:21.420377 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:19:21.507376 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:19:21.516180 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:19:21.535610 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:19:21.560086 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:19:21.562907 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:19:21.584513 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:19:21.596774 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:19:21.639200 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:19:21.652467 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:19:21.673301 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:21.682439 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:19:21.686409 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:19:21.692919 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:19:21.693286 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:19:21.696979 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:19:21.697654 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:19:21.724606 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:19:21.756678 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:19:21.792520 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:19:21.843230 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:19:21.843819 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:19:21.859943 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:19:21.892735 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:19:21.901355 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:19:21.932317 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:19:21.967926 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:19:21.969208 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:19:21.974210 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:21.985330 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:19:22.044231 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:19:22.045848 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:19:22.064553 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:22.077338 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:19:22.109138 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:19:22.223697 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:19:22.270049 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:19:22.274803 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:19:22.323238 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:19:22.382040 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:19:22.397524 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:19:22.462538 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:19:22.497720 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:19:22.505220 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:19:22.516182 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:19:22.546231 27916 raft_consensus.cc:482] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:19:22.546495 27916 raft_consensus.cc:504] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4cfb8fc5a474177af6848a6efa13a27" member_type: VOTER last_known_addr { host: "127.19.94.196" port: 32843 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } }
I20240502 14:19:22.547500 27916 leader_election.cc:288] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:19:22.555382 26755 leader_election.cc:334] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:22.555711 26754 leader_election.cc:334] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:19:22.555845 26754 leader_election.cc:302] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4cfb8fc5a474177af6848a6efa13a27; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:19:22.557020 27514 raft_consensus.cc:2726] T c32ea761f2834d758529890e5b496f88 P e4cfb8fc5a474177af6848a6efa13a27 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:19:22.557706 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:19:22.587388 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:19:22.596730 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:19:22.606287 26493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 168 similar messages]
I20240502 14:19:22.614601 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:19:22.644639 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:22.647965 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:19:22.676360 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:19:22.680030 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:19:22.700078 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:19:22.729197 26754 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 143 similar messages]
W20240502 14:19:22.739490 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:19:22.758504 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:19:22.760747 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:19:22.762120 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:19:22.764658 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:19:22.913594 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:19:22.927901 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:19:22.994033 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:19:23.040437 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:19:23.062196 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:19:23.072852 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:19:23.084157 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:23.090519 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:19:23.148293 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:19:23.149893 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:23.154140 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:19:23.161295 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:19:23.179661 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:19:23.183626 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:19:23.192763 26362 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 94 similar messages]
I20240502 14:19:23.193883 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:19:23.195794 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:19:23.204707 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:19:23.218580 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:19:23.249899 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:19:23.270505 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:19:23.298003 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:23.309139 29511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111) [suppressed 137 similar messages]
W20240502 14:19:23.312264 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:19:23.396457 26624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240502 14:19:23.397362 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:19:23.406771 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:19:23.418788 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:19:23.436308 29111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240502 14:19:23.500903 29511 consensus_peers.cc:485] T 0c7ccae60f58427fbd39c26b98817963 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:19:23.508196 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:19:23.544116 26006 raft_consensus.cc:482] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:19:23.544358 26006 raft_consensus.cc:504] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } }
I20240502 14:19:23.545311 26006 leader_election.cc:288] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659), 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781)
W20240502 14:19:23.551337 26493 leader_election.cc:334] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:23.564955 26494 leader_election.cc:334] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:19:23.565124 26494 leader_election.cc:302] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:19:23.565356 26006 raft_consensus.cc:2726] T 457516f59dee4436b474af63819fad4e P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:19:23.592797 26754 consensus_peers.cc:485] T e45de4b8415446a08e9550f911367624 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:19:23.593389 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:23.640714 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:19:23.645983 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:19:23.693661 26006 raft_consensus.cc:482] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 68a3c2b17ed94e06aba66ed4178132ab)
I20240502 14:19:23.693836 26006 raft_consensus.cc:504] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba187e38a2754d60a12de9ac4856430a" member_type: VOTER last_known_addr { host: "127.19.94.194" port: 36025 } } peers { permanent_uuid: "68a3c2b17ed94e06aba66ed4178132ab" member_type: VOTER last_known_addr { host: "127.19.94.199" port: 41781 } } peers { permanent_uuid: "c1100e16b3b9495ca61d4f95e89b6a89" member_type: VOTER last_known_addr { host: "127.19.94.198" port: 43659 } }
W20240502 14:19:23.694049 29111 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 14:19:23.694553 26006 leader_election.cc:288] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781), c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659)
W20240502 14:19:23.696385 26494 leader_election.cc:334] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:23.697474 26493 leader_election.cc:334] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
I20240502 14:19:23.697577 26493 leader_election.cc:302] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba187e38a2754d60a12de9ac4856430a; no voters: 68a3c2b17ed94e06aba66ed4178132ab, c1100e16b3b9495ca61d4f95e89b6a89
I20240502 14:19:23.698089 26006 raft_consensus.cc:2726] T b8430c13474b4850b71020c4a89bea68 P ba187e38a2754d60a12de9ac4856430a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:19:23.709381 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:19:23.715080 29511 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.94.196:32843, user_credentials={real_user=slave}} blocked reactor thread for 47611.7us
W20240502 14:19:23.852455 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
I20240502 14:19:23.854524 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:19:23.883746 29511 consensus_peers.cc:485] T 1704fbfbeed04dcaac8bc663b7ee2a5b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:19:23.896281 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:23.922760 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:19:23.944118 29112 consensus_peers.cc:485] T 177a34c96aa84cb4aa50bbd79cab4144 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:19:23.948709 26755 consensus_peers.cc:485] T 4df2bd4e182c4fe991da385d92d3f76f P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:19:23.977560 26755 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:19:23.982327 29511 consensus_peers.cc:485] T 2dc3b47b4ab145ccb9efbf1b5740c7f6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:19:23.982519 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:19:24.005123 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 14:19:24.006155 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:24.014011 26493 consensus_peers.cc:485] T 310aa6ad3604487589dbe96afa25bb05 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 14:19:24.018244 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:19:24.027457 26625 consensus_peers.cc:485] T 451b303a32a3404d8d584203d6704b35 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:19:24.055912 29510 consensus_peers.cc:485] T c115987981694996abe797e03b27f799 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:19:24.103535 26362 consensus_peers.cc:485] T 6814613d36a74353a99673cf16aa9821 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:19:24.106289 26754 consensus_peers.cc:485] T 40aa23be0caa427e89e8fdaf905dc052 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:19:24.116261 26493 consensus_peers.cc:485] T 56285a2fdcdd47e9b9c81b4b8993c4aa P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:19:24.127733 26754 consensus_peers.cc:485] T abdd0f01d8d5453793c7e9461af10516 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:19:24.127955 26755 consensus_peers.cc:485] T be1c8c53ff6843a9bf07d006170f1f57 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:19:24.130807 29511 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:19:24.180537 26754 consensus_peers.cc:485] T 6b4745b1153b4acd999ac3a5c137ff53 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:19:24.226976 26362 consensus_peers.cc:485] T bb3e4ec75888419aa4e065cf5513ae5b P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:19:24.290105 26625 consensus_peers.cc:485] T 870100caa8bf4ca1b49337ba374b1f83 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:19:24.364720 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:24.380373 26493 consensus_peers.cc:485] T 9d28bfbd29c94d2ebbedbae53231bf1e P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:19:24.383966 26493 consensus_peers.cc:485] T ebbe2bc571aa44d0a093819144ce49b1 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:19:24.424134 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:24.428999 26494 consensus_peers.cc:485] T 01e6bc3689db46e2ad7d0484a29af287 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:19:24.438588 26493 consensus_peers.cc:485] T 33a3639c26bf4b35a6e15d6ef173751c P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:19:24.510155 26494 consensus_peers.cc:485] T 4bf6e7410afa4c048f5842ec2a426d77 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:19:24.511340 29511 consensus_peers.cc:485] T 83dd34749ff8496483365b30a86db71f P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:19:24.516938 29511 consensus_peers.cc:485] T 82e51aa5371e4731bc00daafc65c7a88 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:19:24.602047 26625 consensus_peers.cc:485] T cf52d0f868fa45418af48639e12f3282 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:19:24.606163 29511 consensus_peers.cc:485] T f3f01a632ddf4810bd6572ff3e722a4b P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:19:24.654645 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:19:24.678148 29511 consensus_peers.cc:485] T 9671eeb3b0e44730a4b0c2849e6ee012 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:19:24.689582 29511 consensus_peers.cc:485] T 01b1a1d502874a3791af520982ecacc0 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:19:24.692826 26754 consensus_peers.cc:485] T c9fb1f7eb71c42d2b8f0f6aacd2b0aeb P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:19:24.696346 26625 consensus_peers.cc:485] T 2108a04310804945b368bcb5f88170e2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:19:24.701375 26363 consensus_peers.cc:485] T 822e90c09531419e9b32af0438fceb57 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:19:24.708334 26493 consensus_peers.cc:485] T 06283c819eaf404b89d3c93c2b080a70 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:19:24.730618 29111 consensus_peers.cc:485] T 46aaa6186d254ff7bd493bd85479015d P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 14:19:24.745286 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:19:24.828552 26624 consensus_peers.cc:485] T 924ce5625bf34a739169b005ee4bedc2 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:19:24.829181 26755 consensus_peers.cc:485] T 533b42de2de14abc96c00a928d5deed1 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:19:24.861353 29111 consensus_peers.cc:485] T f4973d1d77904cb9a3cf8d49346b330e P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 14:19:24.872556 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:19:24.882826 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:19:24.898470 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:19:24.899871 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:24.922183 26493 consensus_peers.cc:485] T b0316714f34d44a5b537cdf20af42ce9 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:19:24.922318 26493 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 14:19:24.937567 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:19:24.950265 26493 consensus_peers.cc:485] T 946239523c0646f1835d7a85b4e7d117 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:19:24.971344 29112 consensus_peers.cc:485] T 5e66bc4ba1f84b50884d6492edb1e0d2 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:19:25.000766 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:25.141536 26624 consensus_peers.cc:485] T 2e59791e95cc452ea30d1ed8bb6178ae P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:19:25.145448 26625 consensus_peers.cc:485] T ce03ee354c54457487ea939301ccd90e P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:19:25.153658 26494 consensus_peers.cc:485] T 9f3cbedfa1da42698615a544d9dc3276 P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:19:25.210114 26493 consensus_peers.cc:485] T 66554eb8b44145d4b474707b061c9659 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:19:25.215447 29511 consensus_peers.cc:485] T 736ed8faceec4f0fa07c62952ea32bf6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:19:25.272790 26625 consensus_peers.cc:485] T c41522c60f3942aa9f28f514dcad4e49 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:19:25.285182 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:25.353011 26362 consensus_peers.cc:485] T d6c8fa23fffe491fbfcec03afdb2052a P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:19:25.404940 26362 consensus_peers.cc:485] T e1bdebd770244537a0119baa4e3f8091 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:19:25.446790 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:25.470599 29112 consensus_peers.cc:485] T 47e87174282541acb60a41a32766dee4 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:19:25.480424 26406 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:19:25.485129 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:19:25.500662 26494 consensus_peers.cc:485] T f94a6ca24bf748c1ac4f12df761d537d P ba187e38a2754d60a12de9ac4856430a -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:19:25.514185 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:19:25.524137 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:25.554026 26538 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 14:19:25.585625 26754 consensus_peers.cc:485] T dc3740d0671d42bc90eb9baeb02f5b0a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:19:25.641427 26798 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:19:25.679821 26363 consensus_peers.cc:485] T eb065120d1304a2a81ce932934b911f3 P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:19:25.687629 26362 consensus_peers.cc:485] T 13136deb9046440f88bf3471ff42312c P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:19:25.700922 26362 consensus_peers.cc:485] T c9c7cac98ef14eab82e7a6f4d4d21f51 P 89af7f423cf44717a08d6b142f0ed49d -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:19:25.750008 26493 consensus_peers.cc:485] T d7f02f897ce54fefb03832da3c2d95e0 P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:19:25.782742 26624 consensus_peers.cc:485] T d0838f171f3146a181ff1f3e843cdec1 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:19:25.802151 29111 consensus_peers.cc:485] T 2434830a866e4f7dbe7c8fa18d48e735 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:19:25.854786 26493 consensus_peers.cc:485] T 3f6d55d89e55425183da73dbb3f4175f P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:19:25.860292 26493 consensus_peers.cc:485] T 842d4e7e5ee74e309fb7eba6d7b7ae8b P ba187e38a2754d60a12de9ac4856430a -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:19:25.871654 26755 consensus_peers.cc:485] T a42e68ab03f84d4b92893006fcf6f229 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:19:25.898375 26755 consensus_peers.cc:462] T 39a61744642f40c8ae92da708dc91ca3 P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111)
W20240502 14:19:25.904501 26363 consensus_peers.cc:485] T 112821e7380e41dcac66a38e21ba809d P 89af7f423cf44717a08d6b142f0ed49d -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:19:25.934981 29111 consensus_peers.cc:485] T 253449163c65471d8736b92b25da0f92 P 1cb67158133e4648a6eceba32d4dbba3 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:19:25.962236 29510 consensus_peers.cc:485] T f18990f33bf74c31a0faf5802cc1856d P 4092902b3e674857a0147f1b4f2cdb86 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Couldn't send request to peer c1100e16b3b9495ca61d4f95e89b6a89. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:19:25.981554 26624 consensus_peers.cc:462] T 0290ebbcf609427cae8fce6637019336 P 69ab78b2a1b74d2bbf2cdeafa0a4af23 -> Peer c1100e16b3b9495ca61d4f95e89b6a89 (127.19.94.198:43659): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.94.198:43659: connect: Connection refused (error 111)
W20240502 14:19:26.021548 29511 consensus_peers.cc:485] T 0c1db9ae649f4ac0bbe05fd07eba7df6 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:19:26.025696 26755 consensus_peers.cc:485] T ffaded0bc6204e8c8576ea8eabdd18ab P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:19:26.044775 29554 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:19:26.045869 26669 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:19:26.053170 29155 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:19:26.057487 26755 consensus_peers.cc:485] T 19f81d8b28564c0dbb0b80bd5018ed5a P e4cfb8fc5a474177af6848a6efa13a27 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn't send request to peer 68a3c2b17ed94e06aba66ed4178132ab. Status: Network error: Client connection negotiation failed: client connection to 127.19.94.199:41781: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:19:26.060175 29511 consensus_peers.cc:485] T 23f576e1bede478496122a9d1a2e7515 P 4092902b3e674857a0147f1b4f2cdb86 -> Peer 68a3c2b17ed94e06aba66ed4178132ab (127.19.94.199:41781): Couldn'